JUnit version 4.13.2
.18:29:09.379 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1783535349371
18:29:09.441 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
18:29:09.478 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
18:29:09.492 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-tasknrBYU1/test-tmp
18:29:09.519 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/asan/bin
18:29:09.522 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-tasknrBYU1/java/build/dist-test/../../../build/asan/bin/kudu, test, mini_cluster, --serialization=pb]
18:29:09.719 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

18:29:09.902 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
18:29:09.903 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

18:29:09.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:09.907711 20702 mini_oidc.cc:150] Starting JWKS server
18:29:09.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:09.910753 20702 webserver.cc:534] Webserver started at http://127.0.0.1:38163/ using document root <none> and password file <none>
18:29:09.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:09.911100 20702 mini_oidc.cc:186] Starting OIDC Discovery server
18:29:09.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:09.911904 20702 webserver.cc:534] Webserver started at http://127.0.0.1:42479/ using document root <none> and password file <none>
18:29:09.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T18:29:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
18:29:09.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T18:29:09Z Disabled control of system clock
18:29:09.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:09.941009 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:09.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:09.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/wal
18:29:09.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/data
18:29:09.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:09.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:09.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:09.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/logs
18:29:09.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/data/info.pb
18:29:09.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:09.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:09.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:09.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:09.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:09.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
18:29:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:09.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:29:09.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:29:09.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.190:43499
18:29:09.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.190
18:29:09.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:09.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:09.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:09.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:09.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:09.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:29:09.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:09.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:29:09.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38163/jwks.json with env {}
18:29:10.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.258211 20713 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:10.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.258621 20713 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:10.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.258744 20713 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:10.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.267473 20713 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
18:29:10.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.267585 20713 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:10.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.267663 20713 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
18:29:10.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.267720 20713 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
18:29:10.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.267825 20713 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
18:29:10.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.267879 20713 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:38163/jwks.json
18:29:10.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.278218 20713 master_runner.cc:387] Master server non-default flags:
18:29:10.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:10.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:10.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:10.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:10.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/data
18:29:10.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/wal
18:29:10.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:29:10.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:10.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:10.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:10.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:29:10.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.190:43499
18:29:10.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:29:10.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:10.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:29:10.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38163/jwks.json
18:29:10.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:10.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:10.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/data/info.pb
18:29:10.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.190
18:29:10.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:10.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:10.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:10.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:10.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:10.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:10.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/logs
18:29:10.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:10.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:10.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:10.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
18:29:10.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:10.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:10.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:10.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:10.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:10.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:10.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.280546 20713 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:10.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.283177 20713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:10.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.292991 20719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:10.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.292970 20718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:10.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.292939 20721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:10.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.294579 20713 server_base.cc:1061] running on GCE node
18:29:10.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.295286 20713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:10.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.296677 20713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:10.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.297884 20713 hybrid_clock.cc:648] HybridClock initialized: now 1783535350297779 us; error 98 us; skew 500 ppm
18:29:10.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.298188 20713 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:10.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.300338 20713 webserver.cc:534] Webserver started at http://127.20.55.190:43985/ using document root <none> and password file <none>
18:29:10.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.301011 20713 fs_manager.cc:362] Metadata directory not provided
18:29:10.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.301098 20713 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:10.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.301592 20713 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:29:10.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.303890 20713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/data/instance:
18:29:10.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e7e5efb1bab346d6bafcb31e0402cf2a"
18:29:10.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:10 on dist-test-slave-4hqz"
18:29:10.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.304742 20713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/wal/instance:
18:29:10.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e7e5efb1bab346d6bafcb31e0402cf2a"
18:29:10.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:10 on dist-test-slave-4hqz"
18:29:10.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.309934 20713 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
18:29:10.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.313460 20727 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:10.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.315068 20713 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
18:29:10.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.315263 20713 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/wal
18:29:10.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e7e5efb1bab346d6bafcb31e0402cf2a"
18:29:10.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:10 on dist-test-slave-4hqz"
18:29:10.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.315636 20713 fs_report.cc:389] FS layout report
18:29:10.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:10.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/wal
18:29:10.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/wal
18:29:10.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/data/data
18:29:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:10.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:10.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:10.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:10.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.356148 20713 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:10.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.357268 20713 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:10.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.357616 20713 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:10.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.377450 20713 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.190:43499
18:29:10.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.377470 20778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.190:43499 every 8 connection(s)
18:29:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.379222 20713 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/data/info.pb
18:29:10.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.383643 20779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
18:29:10.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.387931 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 20713
18:29:10.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.388259 20702 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-0/wal/instance
18:29:10.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.391222 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:10.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:10.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/wal
18:29:10.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/data
18:29:10.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:10.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:10.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:10.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:10.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:10.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:10.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/logs
18:29:10.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/data/info.pb
18:29:10.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:10.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:10.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:10.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:10.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
18:29:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:10.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:29:10.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:29:10.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.189:45691
18:29:10.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.189
18:29:10.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:10.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:10.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:10.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:10.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:10.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:29:10.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:10.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:29:10.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38163/jwks.json with env {}
18:29:10.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.390668 20779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } has no permanent_uuid. Determining permanent_uuid...
18:29:10.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.413578 20779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } has no permanent_uuid. Determining permanent_uuid...
18:29:10.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.414736 20728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.189:45691: connect: Connection refused (error 111)
18:29:10.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.416371 20779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.189:45691: Network error: Client connection negotiation failed: client connection to 127.20.55.189:45691: connect: Connection refused (error 111)
18:29:10.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.465692 20779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } attempt: 1
18:29:10.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.467875 20779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.189:45691: Network error: Client connection negotiation failed: client connection to 127.20.55.189:45691: connect: Connection refused (error 111)
18:29:10.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.536160 20779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } attempt: 2
18:29:10.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.538420 20779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.189:45691: Network error: Client connection negotiation failed: client connection to 127.20.55.189:45691: connect: Connection refused (error 111)
18:29:10.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.629698 20779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } attempt: 3
18:29:10.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.632228 20779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.189:45691: Network error: Client connection negotiation failed: client connection to 127.20.55.189:45691: connect: Connection refused (error 111)
18:29:10.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.720140 20783 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:10.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.720471 20783 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:10.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.720592 20783 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.729154 20783 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
18:29:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.729271 20783 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.729354 20783 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
18:29:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.729413 20783 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
18:29:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.729497 20783 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
18:29:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.729548 20783 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:38163/jwks.json
18:29:10.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.739550 20783 master_runner.cc:387] Master server non-default flags:
18:29:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/data
18:29:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/wal
18:29:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:29:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:29:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.189:45691
18:29:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:29:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:29:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38163/jwks.json
18:29:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/data/info.pb
18:29:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.189
18:29:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:10.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:10.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/logs
18:29:10.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:10.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:10.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:10.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
18:29:10.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:10.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:10.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:10.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:10.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:10.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:10.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.741707 20783 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:10.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.744292 20783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:10.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.753016 20790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:10.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.753041 20789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:10.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.755055 20792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:10.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.755589 20783 server_base.cc:1061] running on GCE node
18:29:10.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.756446 20783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:10.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.757818 20783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:10.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.759070 20783 hybrid_clock.cc:648] HybridClock initialized: now 1783535350759016 us; error 40 us; skew 500 ppm
18:29:10.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.759491 20783 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:10.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.762084 20783 webserver.cc:534] Webserver started at http://127.20.55.189:43589/ using document root <none> and password file <none>
18:29:10.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.762961 20783 fs_manager.cc:362] Metadata directory not provided
18:29:10.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.763087 20783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:10.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.763507 20783 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:29:10.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.766742 20783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/data/instance:
18:29:10.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e1ee748ecdb649258482e4b993fc3d3e"
18:29:10.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:10 on dist-test-slave-4hqz"
18:29:10.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.767711 20783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/wal/instance:
18:29:10.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e1ee748ecdb649258482e4b993fc3d3e"
18:29:10.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:10 on dist-test-slave-4hqz"
18:29:10.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.773890 20783 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.005s
18:29:10.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.776023 20779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } attempt: 4
18:29:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.777402 20798 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:10.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.778225 20779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.189:45691: Network error: Client connection negotiation failed: client connection to 127.20.55.189:45691: connect: Connection refused (error 111)
18:29:10.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.779139 20783 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
18:29:10.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.779273 20783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/wal
18:29:10.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e1ee748ecdb649258482e4b993fc3d3e"
18:29:10.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:10 on dist-test-slave-4hqz"
18:29:10.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.779440 20783 fs_report.cc:389] FS layout report
18:29:10.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/wal
18:29:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/wal
18:29:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/data/data
18:29:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:10.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.798205 20783 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:10.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.799086 20783 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:10.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.799389 20783 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:10.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.819584 20850 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.189:45691 every 8 connection(s)
18:29:10.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.819607 20783 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.189:45691
18:29:10.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.821286 20783 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/data/info.pb
18:29:10.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.825579 20851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
18:29:10.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.829921 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 20783
18:29:10.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.830065 20702 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-1/wal/instance
18:29:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.831274 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/wal
18:29:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/data
18:29:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/logs
18:29:10.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/data/info.pb
18:29:10.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:10.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:10.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:10.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:10.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:10.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
18:29:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:29:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:29:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.188:35241
18:29:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.188
18:29:10.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:10.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:10.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:10.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:10.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:10.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:29:10.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:10.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:29:10.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38163/jwks.json with env {}
18:29:10.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.831307 20851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } has no permanent_uuid. Determining permanent_uuid...
18:29:10.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.848254 20851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } has no permanent_uuid. Determining permanent_uuid...
18:29:10.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.857105 20851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } has no permanent_uuid. Determining permanent_uuid...
18:29:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.858184 20799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.188:35241: connect: Connection refused (error 111)
18:29:10.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.859776 20851 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.188:35241: Network error: Client connection negotiation failed: client connection to 127.20.55.188:35241: connect: Connection refused (error 111)
18:29:10.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.909089 20851 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } attempt: 1
18:29:10.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.911175 20851 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.188:35241: Network error: Client connection negotiation failed: client connection to 127.20.55.188:35241: connect: Connection refused (error 111)
18:29:10.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:10.979398 20851 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } attempt: 2
18:29:10.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:10.981539 20851 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.188:35241: Network error: Client connection negotiation failed: client connection to 127.20.55.188:35241: connect: Connection refused (error 111)
18:29:11.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.072849 20851 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } attempt: 3
18:29:11.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.075134 20851 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.188:35241: Network error: Client connection negotiation failed: client connection to 127.20.55.188:35241: connect: Connection refused (error 111)
18:29:11.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.077466 20779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } attempt: 5
18:29:11.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.084566 20779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } has no permanent_uuid. Determining permanent_uuid...
18:29:11.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.086417 20779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.188:35241: Network error: Client connection negotiation failed: client connection to 127.20.55.188:35241: connect: Connection refused (error 111)
18:29:11.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.137657 20779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } attempt: 1
18:29:11.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.139803 20779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.188:35241: Network error: Client connection negotiation failed: client connection to 127.20.55.188:35241: connect: Connection refused (error 111)
18:29:11.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.167932 20853 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:11.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.168303 20853 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:11.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.168385 20853 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.176584 20853 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
18:29:11.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.176676 20853 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:11.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.176730 20853 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
18:29:11.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.176770 20853 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
18:29:11.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.176832 20853 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
18:29:11.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.176867 20853 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:38163/jwks.json
18:29:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.187093 20853 master_runner.cc:387] Master server non-default flags:
18:29:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:11.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:11.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:11.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/data
18:29:11.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/wal
18:29:11.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:29:11.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:11.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:11.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:11.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:29:11.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.188:35241
18:29:11.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:29:11.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:11.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:29:11.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38163/jwks.json
18:29:11.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:11.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:11.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/data/info.pb
18:29:11.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.188
18:29:11.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:11.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:11.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:11.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:11.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:11.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:11.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/logs
18:29:11.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:11.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:11.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:11.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
18:29:11.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:11.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:11.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:11.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:11.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:11.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:11.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.188993 20853 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:11.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.191367 20853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:11.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.201289 20861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:11.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.201289 20862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:11.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.202104 20864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:11.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.202853 20853 server_base.cc:1061] running on GCE node
18:29:11.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.203598 20853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:11.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.204972 20853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:11.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.206187 20853 hybrid_clock.cc:648] HybridClock initialized: now 1783535351206122 us; error 53 us; skew 500 ppm
18:29:11.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.206553 20853 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:11.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.208025 20779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } attempt: 2
18:29:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.208861 20853 webserver.cc:534] Webserver started at http://127.20.55.188:41623/ using document root <none> and password file <none>
18:29:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.209717 20853 fs_manager.cc:362] Metadata directory not provided
18:29:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.209842 20853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.210110 20779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.188:35241: Network error: Client connection negotiation failed: client connection to 127.20.55.188:35241: connect: Connection refused (error 111)
18:29:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.210295 20853 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:29:11.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.212683 20853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/data/instance:
18:29:11.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2886b90f8c324fd791d9877df321df05"
18:29:11.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:11 on dist-test-slave-4hqz"
18:29:11.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.213493 20853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/wal/instance:
18:29:11.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2886b90f8c324fd791d9877df321df05"
18:29:11.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:11 on dist-test-slave-4hqz"
18:29:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.218407 20851 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } attempt: 4
18:29:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.220466 20851 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.55.188:35241: Network error: Client connection negotiation failed: client connection to 127.20.55.188:35241: connect: Connection refused (error 111)
18:29:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.220767 20853 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.008s	sys 0.001s
18:29:11.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.224308 20870 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:11.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.225615 20853 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
18:29:11.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.225842 20853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/wal
18:29:11.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2886b90f8c324fd791d9877df321df05"
18:29:11.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:11 on dist-test-slave-4hqz"
18:29:11.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.226089 20853 fs_report.cc:389] FS layout report
18:29:11.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:11.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/wal
18:29:11.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/wal
18:29:11.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/data/data
18:29:11.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:11.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:11.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:11.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:11.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:11.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:11.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:11.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:11.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:11.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:11.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:11.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.251121 20853 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:11.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.252213 20853 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:11.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.252552 20853 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:11.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.272404 20853 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.188:35241
18:29:11.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.272428 20921 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.188:35241 every 8 connection(s)
18:29:11.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.274112 20853 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/data/info.pb
18:29:11.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.278465 20922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
18:29:11.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.280616 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 20853
18:29:11.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.280814 20702 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/master-2/wal/instance
18:29:11.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.286077 20922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } has no permanent_uuid. Determining permanent_uuid...
18:29:11.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.299604 20922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } has no permanent_uuid. Determining permanent_uuid...
18:29:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.306830 20922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } has no permanent_uuid. Determining permanent_uuid...
18:29:11.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.316306 20779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } attempt: 3
18:29:11.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.318831 20922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05: Bootstrap starting.
18:29:11.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.323418 20922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05: Neither blocks nor log segments found. Creating new log.
18:29:11.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.324826 20922 log.cc:826] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05: Log is configured to *not* fsync() on all Append() calls
18:29:11.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.327390 20779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a: Bootstrap starting.
18:29:11.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.327781 20922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05: No bootstrap required, opened a new log
18:29:11.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.330246 20779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a: Neither blocks nor log segments found. Creating new log.
18:29:11.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.331269 20779 log.cc:826] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a: Log is configured to *not* fsync() on all Append() calls
18:29:11.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.333981 20779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a: No bootstrap required, opened a new log
18:29:11.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.333751 20922 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } }
18:29:11.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.334106 20922 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:29:11.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.334232 20922 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2886b90f8c324fd791d9877df321df05, State: Initialized, Role: FOLLOWER
18:29:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.334985 20922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } }
18:29:11.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.335963 20932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } } }
18:29:11.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.336269 20932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [sys.catalog]: This master's current role is: FOLLOWER
18:29:11.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.337000 20922 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [sys.catalog]: configured and running, proceeding with master startup.
18:29:11.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.340006 20779 raft_consensus.cc:359] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } }
18:29:11.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.340679 20779 raft_consensus.cc:385] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:29:11.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.340821 20779 raft_consensus.cc:740] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e5efb1bab346d6bafcb31e0402cf2a, State: Initialized, Role: FOLLOWER
18:29:11.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.341511 20779 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } }
18:29:11.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.342778 20941 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } } }
18:29:11.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.343113 20941 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [sys.catalog]: This master's current role is: FOLLOWER
18:29:11.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.344170 20779 sys_catalog.cc:565] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [sys.catalog]: configured and running, proceeding with master startup.
18:29:11.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.349387 20944 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
18:29:11.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.349529 20944 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
18:29:11.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.345618 20853 master_runner.cc:187] Error getting master registration for 127.20.55.190:43499: OK, instance_id { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" instance_seqno: 1783535350372169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
18:29:11.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.356541 20853 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
18:29:11.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.360526 20733 tablet.cc:2406] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:29:11.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.367992 20957 catalog_manager.cc:1593] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
18:29:11.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.370046 20957 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
18:29:11.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.366698 20713 master_runner.cc:187] Error getting master registration for 127.20.55.189:45691: OK, instance_id { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" instance_seqno: 1783535350814197 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
18:29:11.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.373383 20713 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
18:29:11.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.504851 20876 tablet.cc:2406] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:29:11.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.519690 20851 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } attempt: 5
18:29:11.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.532389 20851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e: Bootstrap starting.
18:29:11.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.537089 20851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e: Neither blocks nor log segments found. Creating new log.
18:29:11.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.538944 20851 log.cc:826] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e: Log is configured to *not* fsync() on all Append() calls
18:29:11.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.542407 20851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e: No bootstrap required, opened a new log
18:29:11.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.550011 20851 raft_consensus.cc:359] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } }
18:29:11.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.550333 20851 raft_consensus.cc:385] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:29:11.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.550477 20851 raft_consensus.cc:740] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ee748ecdb649258482e4b993fc3d3e, State: Initialized, Role: FOLLOWER
18:29:11.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.551237 20851 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } }
18:29:11.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.552557 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } } }
18:29:11.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.553108 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [sys.catalog]: This master's current role is: FOLLOWER
18:29:11.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.553766 20851 sys_catalog.cc:565] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [sys.catalog]: configured and running, proceeding with master startup.
18:29:11.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.573454 20783 master_runner.cc:264] No leader master found from master e1ee748ecdb649258482e4b993fc3d3e
18:29:11.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.573601 20783 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
18:29:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.577914 20971 catalog_manager.cc:1593] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
18:29:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.578523 20971 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
18:29:11.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.663534 20960 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:29:11.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.663858 20960 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } }
18:29:11.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.665879 20960 leader_election.cc:290] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7e5efb1bab346d6bafcb31e0402cf2a (127.20.55.190:43499), 2886b90f8c324fd791d9877df321df05 (127.20.55.188:35241)
18:29:11.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.667142 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ee748ecdb649258482e4b993fc3d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" is_pre_election: true
18:29:11.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.668277 20897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ee748ecdb649258482e4b993fc3d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2886b90f8c324fd791d9877df321df05" is_pre_election: true
18:29:11.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.668818 20897 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ee748ecdb649258482e4b993fc3d3e in term 0.
18:29:11.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.669513 20799 leader_election.cc:304] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2886b90f8c324fd791d9877df321df05, e1ee748ecdb649258482e4b993fc3d3e; no voters: 
18:29:11.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.671530 20754 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ee748ecdb649258482e4b993fc3d3e in term 0.
18:29:11.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.676230 20960 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [term 0 FOLLOWER]: Leader pre-election won for term 1
18:29:11.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.676406 20960 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
18:29:11.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.676529 20960 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [term 0 FOLLOWER]: Advancing to term 1
18:29:11.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.680042 20960 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } }
18:29:11.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.681169 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ee748ecdb649258482e4b993fc3d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a"
18:29:11.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.681408 20754 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [term 0 FOLLOWER]: Advancing to term 1
18:29:11.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.687399 20754 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ee748ecdb649258482e4b993fc3d3e in term 1.
18:29:11.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.687949 20799 leader_election.cc:304] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1ee748ecdb649258482e4b993fc3d3e, e7e5efb1bab346d6bafcb31e0402cf2a; no voters: 
18:29:11.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.691193 20960 leader_election.cc:290] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [CANDIDATE]: Term 1 election: Requested vote from peers e7e5efb1bab346d6bafcb31e0402cf2a (127.20.55.190:43499), 2886b90f8c324fd791d9877df321df05 (127.20.55.188:35241)
18:29:11.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.692046 20960 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [term 1 FOLLOWER]: Leader election won for term 1
18:29:11.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.692770 20960 raft_consensus.cc:697] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [term 1 LEADER]: Becoming Leader. State: Replica: e1ee748ecdb649258482e4b993fc3d3e, State: Running, Role: LEADER
18:29:11.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.694257 20960 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } }
18:29:11.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.704546 20897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ee748ecdb649258482e4b993fc3d3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2886b90f8c324fd791d9877df321df05"
18:29:11.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.704785 20897 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [term 0 FOLLOWER]: Advancing to term 1
18:29:11.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.708421 20897 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ee748ecdb649258482e4b993fc3d3e in term 1.
18:29:11.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.711634 20973 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ee748ecdb649258482e4b993fc3d3e. Latest consensus state: current_term: 1 leader_uuid: "e1ee748ecdb649258482e4b993fc3d3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } } }
18:29:11.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.716405 20973 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [sys.catalog]: This master's current role is: LEADER
18:29:11.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.724771 20975 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
18:29:11.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.727480 20975 catalog_manager.cc:1519] Initializing Kudu cluster ID...
18:29:11.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.744001 20897 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [term 1 FOLLOWER]: Refusing update from remote peer e1ee748ecdb649258482e4b993fc3d3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:29:11.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.749405 20973 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
18:29:11.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.758512 20754 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [term 1 FOLLOWER]: Refusing update from remote peer e1ee748ecdb649258482e4b993fc3d3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:29:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.760118 20973 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
18:29:11.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.798763 20941 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ee748ecdb649258482e4b993fc3d3e. Latest consensus state: current_term: 1 leader_uuid: "e1ee748ecdb649258482e4b993fc3d3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } } }
18:29:11.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.799067 20941 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [sys.catalog]: This master's current role is: FOLLOWER
18:29:11.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:11.802026 20804 tablet.cc:2406] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:29:11.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.807163 20976 mvcc.cc:204] Tried to move back new op lower bound from 7305360800730189824 to 7305360800552689664. Current Snapshot: MvccSnapshot[applied={T|T < 7305360800730189824}]
18:29:11.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.811231 20978 mvcc.cc:204] Tried to move back new op lower bound from 7305360800730189824 to 7305360800552689664. Current Snapshot: MvccSnapshot[applied={T|T < 7305360800730189824}]
18:29:11.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.820266 20932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ee748ecdb649258482e4b993fc3d3e. Latest consensus state: current_term: 1 leader_uuid: "e1ee748ecdb649258482e4b993fc3d3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } } }
18:29:11.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.820523 20932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [sys.catalog]: This master's current role is: FOLLOWER
18:29:11.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.825433 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1ee748ecdb649258482e4b993fc3d3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } } }
18:29:11.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.825690 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [sys.catalog]: This master's current role is: LEADER
18:29:11.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.831296 20973 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1ee748ecdb649258482e4b993fc3d3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } } }
18:29:11.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.832118 20973 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e [sys.catalog]: This master's current role is: LEADER
18:29:11.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.834388 20941 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1ee748ecdb649258482e4b993fc3d3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } } }
18:29:11.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.834579 20977 mvcc.cc:204] Tried to move back new op lower bound from 7305360800730189824 to 7305360800552689664. Current Snapshot: MvccSnapshot[applied={T|T < 7305360800730189824}]
18:29:11.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.834631 20941 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a [sys.catalog]: This master's current role is: FOLLOWER
18:29:11.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.837844 20975 catalog_manager.cc:1382] Generated new cluster ID: 5bf541eff3f34dbab75010b3952d91aa
18:29:11.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.837947 20975 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
18:29:11.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.846539 20932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1ee748ecdb649258482e4b993fc3d3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5efb1bab346d6bafcb31e0402cf2a" member_type: VOTER last_known_addr { host: "127.20.55.190" port: 43499 } } peers { permanent_uuid: "e1ee748ecdb649258482e4b993fc3d3e" member_type: VOTER last_known_addr { host: "127.20.55.189" port: 45691 } } peers { permanent_uuid: "2886b90f8c324fd791d9877df321df05" member_type: VOTER last_known_addr { host: "127.20.55.188" port: 35241 } } }
18:29:11.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.846763 20932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05 [sys.catalog]: This master's current role is: FOLLOWER
18:29:11.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.871518 20975 catalog_manager.cc:1405] Generated new certificate authority record
18:29:11.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.873641 20975 catalog_manager.cc:1539] Loading token signing keys...
18:29:11.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.895737 20975 catalog_manager.cc:6069] T 00000000000000000000000000000000 P e1ee748ecdb649258482e4b993fc3d3e: Generated new TSK 0
18:29:11.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.896795 20975 catalog_manager.cc:1549] Initializing in-progress tserver states...
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:11.921788 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/logs
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.129:0
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.129
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.129
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:11.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:11.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:12.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.249123 20985 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:12.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.249452 20985 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:12.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.249563 20985 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:12.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.257859 20985 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:12.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.258080 20985 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.129
18:29:12.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.268348 20985 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:12.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.129:0
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.129
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.129
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/logs
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:12.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:12.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:12.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:12.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:12.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:12.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:12.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:12.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:12.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:12.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.270685 20985 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:12.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.272910 20985 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:12.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.286514 20991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:12.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.286494 20990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:12.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.299937 20993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:12.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.300616 20985 server_base.cc:1061] running on GCE node
18:29:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.301344 20985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:12.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.302683 20985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:12.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.304095 20985 hybrid_clock.cc:648] HybridClock initialized: now 1783535352304004 us; error 95 us; skew 500 ppm
18:29:12.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.304390 20985 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:12.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.306671 20985 webserver.cc:534] Webserver started at http://127.20.55.129:39253/ using document root <none> and password file <none>
18:29:12.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.307515 20985 fs_manager.cc:362] Metadata directory not provided
18:29:12.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.307605 20985 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:12.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.308008 20985 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:29:12.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.311200 20985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/instance:
18:29:12.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d202cde0b06b4bff9c06194f4b40ac5d"
18:29:12.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:12.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.311980 20985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal/instance:
18:29:12.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d202cde0b06b4bff9c06194f4b40ac5d"
18:29:12.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:12.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.318970 20985 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.007s	sys 0.000s
18:29:12.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.322144 20999 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:12.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.323655 20985 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.323827 20985 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d202cde0b06b4bff9c06194f4b40ac5d"
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.324005 20985 fs_report.cc:389] FS layout report
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/data
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:12.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.341838 20985 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:12.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.342629 20985 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:12.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.342862 20985 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:12.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.343724 20985 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:12.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.345254 20985 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
18:29:12.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.345341 20985 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
18:29:12.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.345435 20985 ts_tablet_manager.cc:616] Registered 0 tablets
18:29:12.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.345484 20985 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
18:29:12.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.351825 20944 catalog_manager.cc:1294] Loaded cluster ID: 5bf541eff3f34dbab75010b3952d91aa
18:29:12.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.351940 20944 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05: loading cluster ID for follower catalog manager: success
18:29:12.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.355623 20944 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05: acquiring CA information for follower catalog manager: success
18:29:12.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.357728 20944 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 2886b90f8c324fd791d9877df321df05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
18:29:12.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.373920 20957 catalog_manager.cc:1294] Loaded cluster ID: 5bf541eff3f34dbab75010b3952d91aa
18:29:12.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.374035 20957 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a: loading cluster ID for follower catalog manager: success
18:29:12.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.377568 20957 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a: acquiring CA information for follower catalog manager: success
18:29:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.380125 20957 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e7e5efb1bab346d6bafcb31e0402cf2a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
18:29:12.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.381768 20985 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.129:43419
18:29:12.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.381796 21112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.129:43419 every 8 connection(s)
18:29:12.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.383469 20985 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:12.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.391171 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 20985
18:29:12.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.391418 20702 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal/instance
18:29:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.397642 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/logs
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.130:0
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.130
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.130
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:12.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:12.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.407174 21113 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:12.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.408041 21113 heartbeater.cc:461] Registering TS with master...
18:29:12.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.409327 21113 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:12.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.412259 21115 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:12.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.412418 21115 heartbeater.cc:461] Registering TS with master...
18:29:12.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.412478 20887 ts_manager.cc:194] Registered new tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:12.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.413069 21115 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:12.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.414764 20816 ts_manager.cc:194] Registered new tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:12.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.416792 20816 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.129:47927
18:29:12.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.420740 21114 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:12.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.420889 21114 heartbeater.cc:461] Registering TS with master...
18:29:12.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.421262 21114 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:12.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.422830 20744 ts_manager.cc:194] Registered new tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:12.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.735409 21121 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:12.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.735697 21121 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:12.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.735776 21121 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:12.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.744100 21121 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:12.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.744259 21121 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.130
18:29:12.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.753948 21121 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:12.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:12.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:12.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:12.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:12.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data
18:29:12.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:12.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:12.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:12.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.130:0
18:29:12.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.130
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.130
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/logs
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:12.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.755748 21121 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:12.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.758154 21121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:12.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.769133 21127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:12.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.769510 21128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:12.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:12.769929 21130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:12.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.770944 21121 server_base.cc:1061] running on GCE node
18:29:12.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.771471 21121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:12.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.772356 21121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:12.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.773571 21121 hybrid_clock.cc:648] HybridClock initialized: now 1783535352773534 us; error 29 us; skew 500 ppm
18:29:12.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.773866 21121 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.775981 21121 webserver.cc:534] Webserver started at http://127.20.55.130:42109/ using document root <none> and password file <none>
18:29:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.776675 21121 fs_manager.cc:362] Metadata directory not provided
18:29:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.776762 21121 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:12.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.777050 21121 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:29:12.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.779650 21121 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/instance:
18:29:12.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "37ebb4c0266644fa98fa0851dac80a72"
18:29:12.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:12.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.780349 21121 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal/instance:
18:29:12.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "37ebb4c0266644fa98fa0851dac80a72"
18:29:12.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:12.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.785506 21121 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
18:29:12.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.789166 21136 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.790567 21121 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
18:29:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.790701 21121 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "37ebb4c0266644fa98fa0851dac80a72"
18:29:12.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.790861 21121 fs_report.cc:389] FS layout report
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/data
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:12.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:12.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.807633 21121 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:12.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.808439 21121 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:12.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.808661 21121 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:12.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.809538 21121 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:12.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.811137 21121 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
18:29:12.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.811223 21121 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
18:29:12.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.811290 21121 ts_tablet_manager.cc:616] Registered 0 tablets
18:29:12.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.811336 21121 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
18:29:12.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.845359 21121 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.130:33399
18:29:12.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.845412 21248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.130:33399 every 8 connection(s)
18:29:12.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.846798 21121 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:12.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.855098 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 21121
18:29:12.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.855283 20702 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal/instance
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.859917 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/logs
18:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.131:0
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.131
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.131
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:12.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:12.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:12.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:12.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:12.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.865010 21252 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:12.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.865471 21252 heartbeater.cc:461] Registering TS with master...
18:29:12.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.866134 21251 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:12.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.866255 21251 heartbeater.cc:461] Registering TS with master...
18:29:12.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.866730 21252 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:12.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.866863 21251 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:12.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.867442 21250 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:12.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.867573 21250 heartbeater.cc:461] Registering TS with master...
18:29:12.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.867966 21250 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:12.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.869221 20816 ts_manager.cc:194] Registered new tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:12.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.869323 20887 ts_manager.cc:194] Registered new tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:12.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.869361 20744 ts_manager.cc:194] Registered new tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:12.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:12.870219 20816 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.130:59135
18:29:13.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:13.193555 21257 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:13.193852 21257 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:13.193931 21257 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:13.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:13.202028 21257 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:13.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:13.202189 21257 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.131
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.211974 21257 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.131:0
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.131
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.131
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/logs
18:29:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:13.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:13.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:13.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:13.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:13.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:13.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:13.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:13.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:13.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:13.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.213793 21257 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:13.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.216297 21257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:13.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:13.226294 21262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:13.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:13.226445 21263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:13.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:13.227296 21265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:13.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.227865 21257 server_base.cc:1061] running on GCE node
18:29:13.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.228365 21257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:13.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.229323 21257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:13.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.230531 21257 hybrid_clock.cc:648] HybridClock initialized: now 1783535353230478 us; error 49 us; skew 500 ppm
18:29:13.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.230890 21257 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:13.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.233019 21257 webserver.cc:534] Webserver started at http://127.20.55.131:40047/ using document root <none> and password file <none>
18:29:13.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.233772 21257 fs_manager.cc:362] Metadata directory not provided
18:29:13.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.233902 21257 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:13.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.234297 21257 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
18:29:13.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.236676 21257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/instance:
18:29:13.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "58924cf836aa406bbedb44253800ef1e"
18:29:13.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:13 on dist-test-slave-4hqz"
18:29:13.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.237501 21257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal/instance:
18:29:13.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "58924cf836aa406bbedb44253800ef1e"
18:29:13.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:13 on dist-test-slave-4hqz"
18:29:13.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.242857 21257 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
18:29:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.246860 21271 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:13.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.248126 21257 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
18:29:13.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.248328 21257 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:13.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "58924cf836aa406bbedb44253800ef1e"
18:29:13.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:13 on dist-test-slave-4hqz"
18:29:13.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.248577 21257 fs_report.cc:389] FS layout report
18:29:13.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:13.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/data
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:13.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:13.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.265121 21257 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:13.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.266034 21257 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:13.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.266398 21257 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:13.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.267391 21257 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:13.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.269057 21257 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
18:29:13.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.269181 21257 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
18:29:13.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.269284 21257 ts_tablet_manager.cc:616] Registered 0 tablets
18:29:13.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.269361 21257 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
18:29:13.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.304435 21257 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.131:42103
18:29:13.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.304483 21386 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.131:42103 every 8 connection(s)
18:29:13.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.305976 21257 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:13.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.309885 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 21257
18:29:13.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.310086 20702 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal/instance
18:29:13.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.323402 21389 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:13.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.323508 21388 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:13.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.323585 21387 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:13.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.323858 21387 heartbeater.cc:461] Registering TS with master...
18:29:13.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.323902 21389 heartbeater.cc:461] Registering TS with master...
18:29:13.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.323902 21388 heartbeater.cc:461] Registering TS with master...
18:29:13.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.325080 21388 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:13.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.325130 21389 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:13.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.325208 21387 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:13.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.326817 20816 ts_manager.cc:194] Registered new tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:13.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.326861 20887 ts_manager.cc:194] Registered new tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:13.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.327001 20744 ts_manager.cc:194] Registered new tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:13.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.327804 20816 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.131:47897
18:29:13.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.336341 20702 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
18:29:13.338 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
18:29:13.339 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

18:29:13.347 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.20.55.190"
      port: 43499
    }
    bound_http_address {
      host: "127.20.55.190"
      port: 43985
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.20.55.189"
      port: 45691
    }
    bound_http_address {
      host: "127.20.55.189"
      port: 43589
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.20.55.188"
      port: 35241
    }
    bound_http_address {
      host: "127.20.55.188"
      port: 41623
    }
  }
}

18:29:13.350 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

18:29:13.352 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.20.55.129"
      port: 43419
    }
    bound_http_address {
      host: "127.20.55.129"
      port: 39253
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.20.55.130"
      port: 33399
    }
    bound_http_address {
      host: "127.20.55.130"
      port: 42109
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.20.55.131"
      port: 42103
    }
    bound_http_address {
      host: "127.20.55.131"
      port: 40047
    }
  }
}

18:29:13.352 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
18:29:13.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.420027 21115 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:13.532 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:225)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:246)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
18:29:13.589 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
18:29:13.731 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.20.55.188:35241(127.20.55.188:35241)] connecting to peer
18:29:13.779 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.20.55.189:45691(127.20.55.189:45691)] connecting to peer
18:29:13.780 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.20.55.190:43499(127.20.55.190:43499)] connecting to peer
18:29:13.795 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.20.55.189:45691(127.20.55.189:45691)] Successfully connected to peer
18:29:13.795 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.20.55.190:43499(127.20.55.190:43499)] Successfully connected to peer
18:29:13.795 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.20.55.188:35241(127.20.55.188:35241)] Successfully connected to peer
18:29:13.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:13.872380 21252 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:13.891 [DEBUG - kudu-nio-2] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.20.55.190
18:29:13.891 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.20.55.188
18:29:13.893 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.20.55.189
18:29:14.098 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xd28a61a6, L:/127.0.0.1:35408 - R:/127.20.55.189:45691] using SASL/PLAIN
18:29:14.098 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x5375ee62, L:/127.0.0.1:39252 - R:/127.20.55.190:43499] using SASL/PLAIN
18:29:14.099 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x69350685, L:/127.0.0.1:32916 - R:/127.20.55.188:35241] using SASL/PLAIN
18:29:14.136 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
18:29:14.137 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=439), Traces: [0ms] refreshing cache from master, [21ms] Sub RPC ConnectToMaster: sending RPC to server master-127.20.55.188:35241, [77ms] Sub RPC ConnectToMaster: sending RPC to server master-127.20.55.189:45691, [79ms] Sub RPC ConnectToMaster: sending RPC to server master-127.20.55.190:43499, [410ms] Sub RPC ConnectToMaster: received response from server master-127.20.55.190:43499: OK, [410ms] Sub RPC ConnectToMaster: received response from server master-127.20.55.188:35241: OK, [411ms] Sub RPC ConnectToMaster: received response from server master-127.20.55.189:45691: OK, deferred=null) after lookup
18:29:14.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.180189 20816 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35408:
18:29:14.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685"
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:29:14.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
18:29:14.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
18:29:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:14.185540 20816 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
18:29:14.244 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table e5f5127110da47f582b33706473166b5
18:29:14.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.249686 21047 tablet_service.cc:1511] Processing CreateTablet for tablet fac231686ac94760b2e038da9aa2f387 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685 [id=e5f5127110da47f582b33706473166b5]), partition=RANGE (key) PARTITION UNBOUNDED
18:29:14.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.249720 21184 tablet_service.cc:1511] Processing CreateTablet for tablet fac231686ac94760b2e038da9aa2f387 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685 [id=e5f5127110da47f582b33706473166b5]), partition=RANGE (key) PARTITION UNBOUNDED
18:29:14.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.251098 21047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fac231686ac94760b2e038da9aa2f387. 1 dirs total, 0 dirs full, 0 dirs failed
18:29:14.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.251109 21184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fac231686ac94760b2e038da9aa2f387. 1 dirs total, 0 dirs full, 0 dirs failed
18:29:14.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.257165 21321 tablet_service.cc:1511] Processing CreateTablet for tablet fac231686ac94760b2e038da9aa2f387 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685 [id=e5f5127110da47f582b33706473166b5]), partition=RANGE (key) PARTITION UNBOUNDED
18:29:14.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.258576 21321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fac231686ac94760b2e038da9aa2f387. 1 dirs total, 0 dirs full, 0 dirs failed
18:29:14.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.275775 21418 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Bootstrap starting.
18:29:14.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.276976 21419 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap starting.
18:29:14.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.282702 21419 tablet_bootstrap.cc:654] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Neither blocks nor log segments found. Creating new log.
18:29:14.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.283360 21418 tablet_bootstrap.cc:654] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Neither blocks nor log segments found. Creating new log.
18:29:14.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.284482 21419 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Log is configured to *not* fsync() on all Append() calls
18:29:14.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.284789 21418 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Log is configured to *not* fsync() on all Append() calls
18:29:14.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.290706 21418 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: No bootstrap required, opened a new log
18:29:14.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.291380 21418 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Time spent bootstrapping tablet: real 0.016s	user 0.008s	sys 0.003s
18:29:14.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.296782 21420 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Bootstrap starting.
18:29:14.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.303113 21420 tablet_bootstrap.cc:654] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Neither blocks nor log segments found. Creating new log.
18:29:14.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.304093 21419 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: No bootstrap required, opened a new log
18:29:14.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.304598 21419 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Time spent bootstrapping tablet: real 0.028s	user 0.019s	sys 0.004s
18:29:14.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.304951 21420 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Log is configured to *not* fsync() on all Append() calls
18:29:14.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.305167 21418 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:14.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.305791 21418 raft_consensus.cc:385] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:29:14.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.306070 21418 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d202cde0b06b4bff9c06194f4b40ac5d, State: Initialized, Role: FOLLOWER
18:29:14.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.306978 21418 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:14.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.312659 21419 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:14.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.313628 21419 raft_consensus.cc:385] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:29:14.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.313894 21419 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ebb4c0266644fa98fa0851dac80a72, State: Initialized, Role: FOLLOWER
18:29:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.314765 21419 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:14.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.324359 21418 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Time spent starting tablet: real 0.033s	user 0.017s	sys 0.008s
18:29:14.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.329969 21389 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:14.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.339941 21419 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Time spent starting tablet: real 0.035s	user 0.013s	sys 0.007s
18:29:14.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.342316 21420 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: No bootstrap required, opened a new log
18:29:14.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.342664 21420 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Time spent bootstrapping tablet: real 0.046s	user 0.013s	sys 0.007s
18:29:14.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.348855 21420 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:14.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.349166 21420 raft_consensus.cc:385] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:29:14.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.349239 21420 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58924cf836aa406bbedb44253800ef1e, State: Initialized, Role: FOLLOWER
18:29:14.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.349761 21420 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:14.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.351429 21420 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
18:29:14.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:14.351449 21253 tablet.cc:2406] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:29:14.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:14.388443 21116 tablet.cc:2406] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:29:14.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.510253 21425 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:29:14.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.510661 21425 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:14.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.512681 21425 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:14.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.519797 21067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" is_pre_election: true
18:29:14.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.519793 21341 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58924cf836aa406bbedb44253800ef1e" is_pre_election: true
18:29:14.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.520221 21341 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ebb4c0266644fa98fa0851dac80a72 in term 0.
18:29:14.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.520267 21067 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ebb4c0266644fa98fa0851dac80a72 in term 0.
18:29:14.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.520867 21137 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, 58924cf836aa406bbedb44253800ef1e, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 
18:29:14.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.521237 21425 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 0 FOLLOWER]: Leader pre-election won for term 1
18:29:14.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.521394 21425 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
18:29:14.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.521497 21425 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 0 FOLLOWER]: Advancing to term 1
18:29:14.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.523684 21425 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:14.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.524250 21425 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 1 election: Requested vote from peers d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:14.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.524672 21067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d"
18:29:14.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.524730 21341 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58924cf836aa406bbedb44253800ef1e"
18:29:14.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.524861 21341 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 0 FOLLOWER]: Advancing to term 1
18:29:14.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.524856 21067 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 0 FOLLOWER]: Advancing to term 1
18:29:14.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.526988 21341 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ebb4c0266644fa98fa0851dac80a72 in term 1.
18:29:14.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.527046 21067 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ebb4c0266644fa98fa0851dac80a72 in term 1.
18:29:14.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.527385 21137 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, 58924cf836aa406bbedb44253800ef1e; no voters: 
18:29:14.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.527706 21425 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 1 FOLLOWER]: Leader election won for term 1
18:29:14.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.528132 21425 raft_consensus.cc:697] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 1 LEADER]: Becoming Leader. State: Replica: 37ebb4c0266644fa98fa0851dac80a72, State: Running, Role: LEADER
18:29:14.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.528657 21425 consensus_queue.cc:237] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:14.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.534236 20816 catalog_manager.cc:5696] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130). New cstate: current_term: 1 leader_uuid: "37ebb4c0266644fa98fa0851dac80a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } health_report { overall_health: UNKNOWN } } }
18:29:14.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:14.559111 21390 tablet.cc:2406] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:29:14.745 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table e5f5127110da47f582b33706473166b5
18:29:14.762 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:14.770 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299992, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:14.771 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=null) after lookup
18:29:14.783 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:14.786 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] Successfully connected to peer
18:29:14.871 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0x393721d2, L:/127.0.0.1:38530 - R:/127.20.55.130:33399] using TOKEN/null
18:29:14.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.882579 21067 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 1 FOLLOWER]: Refusing update from remote peer 37ebb4c0266644fa98fa0851dac80a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:29:14.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.882580 21341 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 1 FOLLOWER]: Refusing update from remote peer 37ebb4c0266644fa98fa0851dac80a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:29:14.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.883678 21429 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
18:29:14.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.884162 21425 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
18:29:14.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.929152 21434 mvcc.cc:204] Tried to move back new op lower bound from 7305360813587668992 to 7305360812153643008. Current Snapshot: MvccSnapshot[applied={T|T < 7305360813587668992}]
18:29:14.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.932129 21438 mvcc.cc:204] Tried to move back new op lower bound from 7305360813587668992 to 7305360812153643008. Current Snapshot: MvccSnapshot[applied={T|T < 7305360813587668992}]
18:29:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:14.940151 21437 mvcc.cc:204] Tried to move back new op lower bound from 7305360813587668992 to 7305360812153643008. Current Snapshot: MvccSnapshot[applied={T|T < 7305360813587668992}]
18:29:15.045 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:15.047 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.55.130:33399
18:29:15.049 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:29:15.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:15.049551 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 21121
18:29:15.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:15.075619 21244 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:15.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:15.177068 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 21121
18:29:15.196 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:15.199 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] handling channelInactive
18:29:15.200 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state READY due to: connection closed
18:29:15.200 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: connection closed
18:29:15.222 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:15.223 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:15.223 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [4ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [26ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1499979296(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@aa8be7a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3172cbe2 -> wakeup thread Time-limited test)) after lookup
18:29:15.224 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:15.226 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.226 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.226 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.242 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:15.242 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:15.243 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=46), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [4ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [26ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [27ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [30ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [31ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [46ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1499979296(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@aa8be7a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@790177600 after retry RPC@980485653), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3172cbe2 -> wakeup thread Time-limited test -> (continuation of Deferred@790177600 after retry RPC@980485653))) after lookup
18:29:15.244 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:15.246 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.246 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.246 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.265 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:15.266 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:15.266 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [4ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [26ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [27ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [30ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [31ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [46ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [47ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [50ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [50ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [69ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1499979296(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@aa8be7a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@790177600 after retry RPC@980485653) -> (continuation of Deferred@5413114 after retry RPC@1360787414), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3172cbe2 -> wakeup thread Time-limited test -> (continuation of Deferred@790177600 after retry RPC@980485653) -> (continuation of Deferred@5413114 after retry RPC@1360787414))) after lookup
18:29:15.267 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:15.269 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.270 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.270 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.342 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:15.342 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:15.343 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=146), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [4ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [26ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [27ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [30ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [31ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [46ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [47ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [50ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [50ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [69ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [71ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [74ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [74ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [142ms] refreshing cache from master, [142ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [146ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1499979296(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@aa8be7a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@790177600 after retry RPC@980485653) -> (continuation of Deferred@5413114 after retry RPC@1360787414) -> (continuation of Deferred@88973493 after retry RPC@485446337), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3172cbe2 -> wakeup thread Time-limited test -> (continuation of Deferred@790177600 after retry RPC@980485653) -> (continuation of Deferred@5413114 after retry RPC@1360787414) -> (continuation of Deferred@88973493 after retry RPC@485446337))) after lookup
18:29:15.343 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:15.345 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.345 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.346 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.862 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:15.862 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:15.863 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=666), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [4ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [26ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [27ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [30ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [31ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [46ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [47ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [50ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [50ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [69ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [71ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [74ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [74ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [142ms] refreshing cache from master, [142ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [146ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [147ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [149ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [150ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [662ms] refreshing cache from master, [663ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [666ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1499979296(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@aa8be7a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@790177600 after retry RPC@980485653) -> (continuation of Deferred@5413114 after retry RPC@1360787414) -> (continuation of Deferred@88973493 after retry RPC@485446337) -> (continuation of Deferred@146367974 after retry RPC@742486989), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3172cbe2 -> wakeup thread Time-limited test -> (continuation of Deferred@790177600 after retry RPC@980485653) -> (continuation of Deferred@5413114 after retry RPC@1360787414) -> (continuation of Deferred@88973493 after retry RPC@485446337) -> (continuation of Deferred@146367974 after retry RPC@742486989))) after lookup
18:29:15.864 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:15.865 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.865 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:15.865 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:16.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.484427 21453 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ebb4c0266644fa98fa0851dac80a72)
18:29:16.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.484465 21452 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ebb4c0266644fa98fa0851dac80a72)
18:29:16.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.484637 21453 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:16.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.484673 21452 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:16.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.486320 21452 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:16.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.486486 21453 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:16.488682 21274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:16.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.499845 21066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "58924cf836aa406bbedb44253800ef1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" is_pre_election: true
18:29:16.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.500137 21066 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58924cf836aa406bbedb44253800ef1e in term 1.
18:29:16.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:16.500202 21002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:16.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.501147 21273 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58924cf836aa406bbedb44253800ef1e, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 
18:29:16.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:16.501499 21274 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:16.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.501900 21452 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 1 FOLLOWER]: Leader pre-election won for term 2
18:29:16.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.502041 21452 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37ebb4c0266644fa98fa0851dac80a72)
18:29:16.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.502161 21452 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 1 FOLLOWER]: Advancing to term 2
18:29:16.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:16.503026 21002 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:16.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.505432 21452 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:16.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.506064 21452 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 2 election: Requested vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:16.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.506686 21066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "58924cf836aa406bbedb44253800ef1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d"
18:29:16.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.506917 21066 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 1 FOLLOWER]: Advancing to term 2
18:29:16.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:16.507630 21274 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:16.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.508960 21341 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58924cf836aa406bbedb44253800ef1e" is_pre_election: true
18:29:16.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.509219 21341 raft_consensus.cc:2393] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d202cde0b06b4bff9c06194f4b40ac5d in current term 2: Already voted for candidate 58924cf836aa406bbedb44253800ef1e in this term.
18:29:16.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.509665 21066 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58924cf836aa406bbedb44253800ef1e in term 2.
18:29:16.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.509858 21000 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d202cde0b06b4bff9c06194f4b40ac5d; no voters: 37ebb4c0266644fa98fa0851dac80a72, 58924cf836aa406bbedb44253800ef1e
18:29:16.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.510085 21273 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58924cf836aa406bbedb44253800ef1e, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 37ebb4c0266644fa98fa0851dac80a72
18:29:16.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.510392 21453 raft_consensus.cc:2749] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
18:29:16.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.510510 21452 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 2 FOLLOWER]: Leader election won for term 2
18:29:16.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.510957 21452 raft_consensus.cc:697] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 2 LEADER]: Becoming Leader. State: Replica: 58924cf836aa406bbedb44253800ef1e, State: Running, Role: LEADER
18:29:16.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.511485 21452 consensus_queue.cc:237] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:16.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:16.515204 20816 catalog_manager.cc:5696] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e reported cstate change: term changed from 1 to 2, leader changed from 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130) to 58924cf836aa406bbedb44253800ef1e (127.20.55.131). New cstate: current_term: 2 leader_uuid: "58924cf836aa406bbedb44253800ef1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } health_report { overall_health: HEALTHY } } }
18:29:17.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:17.034562 21274 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e -> Peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Couldn't send request to peer 37ebb4c0266644fa98fa0851dac80a72. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:29:17.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.050575 21066 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 2 FOLLOWER]: Refusing update from remote peer 58924cf836aa406bbedb44253800ef1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
18:29:17.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.051260 21475 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
18:29:17.461 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:17.462 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:17.463 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2266), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [4ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [26ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [27ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [30ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [31ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [46ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [47ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [50ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [50ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [69ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [71ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [74ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [74ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [142ms] refreshing cache from master, [142ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [146ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [147ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [149ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [150ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [662ms] refreshing cache from master, [663ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [666ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [667ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [669ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [669ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2262ms] refreshing cache from master, [2262ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [2265ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1499979296(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@aa8be7a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@790177600 after retry RPC@980485653) -> (continuation of Deferred@5413114 after retry RPC@1360787414) -> (continuation of Deferred@88973493 after retry RPC@485446337) -> (continuation of Deferred@146367974 after retry RPC@742486989) -> (continuation of Deferred@90027303 after retry RPC@581263706), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3172cbe2 -> wakeup thread Time-limited test -> (continuation of Deferred@790177600 after retry RPC@980485653) -> (continuation of Deferred@5413114 after retry RPC@1360787414) -> (continuation of Deferred@88973493 after retry RPC@485446337) -> (continuation of Deferred@146367974 after retry RPC@742486989) -> (continuation of Deferred@90027303 after retry RPC@581263706))) after lookup
18:29:17.463 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:17.464 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] Successfully connected to peer
18:29:17.486 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x91e41b2b, L:/127.0.0.1:34668 - R:/127.20.55.131:42103] using TOKEN/null
18:29:17.507 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.55.130:33399
18:29:17.508 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.509541 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data
18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/logs
18:29:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.130:33399
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.130
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42109
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.130
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:17.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:17.834951 21487 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:17.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:17.835242 21487 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:17.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:17.835325 21487 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:17.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:17.843533 21487 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:17.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:17.843684 21487 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.130
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.853947 21487 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.130:33399
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.130
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42109
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.130
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/logs
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.855520 21487 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:17.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.857316 21487 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:17.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:17.866326 21493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:17.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:17.866417 21492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:17.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:17.867275 21495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:17.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.867668 21487 server_base.cc:1061] running on GCE node
18:29:17.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.868194 21487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:17.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.869061 21487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:17.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.870241 21487 hybrid_clock.cc:648] HybridClock initialized: now 1783535357870191 us; error 42 us; skew 500 ppm
18:29:17.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.870565 21487 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:17.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.872596 21487 webserver.cc:534] Webserver started at http://127.20.55.130:42109/ using document root <none> and password file <none>
18:29:17.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.873266 21487 fs_manager.cc:362] Metadata directory not provided
18:29:17.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.873363 21487 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:17.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.878291 21487 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
18:29:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.881031 21501 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:17.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.882261 21487 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.882468 21487 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "37ebb4c0266644fa98fa0851dac80a72"
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.883160 21487 fs_report.cc:389] FS layout report
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/data
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:17.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:17.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.905592 21487 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:17.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.906555 21487 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.906927 21487 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:17.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.907958 21487 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:17.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.910013 21508 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:29:17.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.916002 21487 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:29:17.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.916103 21487 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
18:29:17.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.916237 21487 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:29:17.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.919210 21487 ts_tablet_manager.cc:616] Registered 1 tablets
18:29:17.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.919327 21487 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
18:29:17.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.919737 21508 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap starting.
18:29:17.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.941694 21508 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Log is configured to *not* fsync() on all Append() calls
18:29:17.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.955826 21508 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:29:17.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.956326 21508 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap complete.
18:29:17.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.957732 21508 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Time spent bootstrapping tablet: real 0.038s	user 0.021s	sys 0.013s
18:29:17.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.965162 21508 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:17.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.966143 21508 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ebb4c0266644fa98fa0851dac80a72, State: Initialized, Role: FOLLOWER
18:29:17.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.967103 21508 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:17.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.967950 21487 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.130:33399
18:29:17.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.968232 21617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.130:33399 every 8 connection(s)
18:29:17.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.968900 21508 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Time spent starting tablet: real 0.011s	user 0.013s	sys 0.000s
18:29:17.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.969491 21487 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:17.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:17.975718 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 21487
18:29:17.976 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:18.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.000141 21620 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:18.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.000545 21620 heartbeater.cc:461] Registering TS with master...
18:29:18.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.001374 21620 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:18.003 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:18.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.002147 21619 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:18.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.002291 21619 heartbeater.cc:461] Registering TS with master...
18:29:18.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.002888 21619 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:18.004 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.55.131:42103
18:29:18.004 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:29:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.005106 20743 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.005229 20816 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:18.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.006412 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 21257
18:29:18.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.006458 21618 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:18.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.006582 21618 heartbeater.cc:461] Registering TS with master...
18:29:18.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.006714 20816 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.130:34625
18:29:18.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.007035 21618 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:18.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.009187 20887 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.022262 21382 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:18.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:18.122035 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 21257
18:29:18.141 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] handling channelInactive
18:29:18.142 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state READY due to: connection closed
18:29:18.142 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:18.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:18.142513 21633 negotiation.cc:336] Failed RPC negotiation. Trace:
18:29:18.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:18.119192 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.55.131:39879 (local address 127.20.55.130:33399)
18:29:18.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:18.123250 (+  4058us) server_negotiation.cc:207] Beginning negotiation
18:29:18.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:18.123256 (+     6us) server_negotiation.cc:400] Waiting for connection header
18:29:18.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:18.141030 (+ 17774us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.55.131:39879: BlockingRecv error: recv got EOF from 127.20.55.131:39879 (error 108)
18:29:18.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":3897,"thread_start_us":3678,"threads_started":1}
18:29:18.143 [DEBUG - Time-limited test] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:18.144 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.144 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.144 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.161 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:18.162 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:18.162 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [2ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [2ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [19ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1554470652(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@66809fc0 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65664a04 -> wakeup thread Time-limited test)) after lookup
18:29:18.163 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:18.163 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.164 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.164 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.181 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:18.181 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:18.181 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [2ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [2ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [19ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [20ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [22ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [22ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [39ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1554470652(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@66809fc0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65664a04 -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130))) after lookup
18:29:18.182 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:18.183 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.183 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.183 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.222 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:18.222 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:18.222 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [2ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [2ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [19ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [20ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [22ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [22ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [39ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [40ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [41ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [41ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [79ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1554470652(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@66809fc0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130) -> (continuation of Deferred@145904798 after retry RPC@160179621), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65664a04 -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130) -> (continuation of Deferred@145904798 after retry RPC@160179621))) after lookup
18:29:18.223 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:18.224 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.224 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.224 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.302 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:18.302 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:18.302 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=160), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [2ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [2ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [19ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [20ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [22ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [22ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [39ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [40ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [41ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [41ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [79ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [81ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [82ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [82ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [157ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [159ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1554470652(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@66809fc0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130) -> (continuation of Deferred@145904798 after retry RPC@160179621) -> (continuation of Deferred@1333456711 after retry RPC@657676800), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65664a04 -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130) -> (continuation of Deferred@145904798 after retry RPC@160179621) -> (continuation of Deferred@1333456711 after retry RPC@657676800))) after lookup
18:29:18.303 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:18.304 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.304 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.304 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.803 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:18.803 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:18.803 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=661), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [2ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [2ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [19ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [20ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [22ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [22ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [39ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [40ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [41ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [41ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [79ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [81ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [82ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [82ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [157ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [159ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [161ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [162ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [162ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [657ms] refreshing cache from master, [657ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [660ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1554470652(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@66809fc0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130) -> (continuation of Deferred@145904798 after retry RPC@160179621) -> (continuation of Deferred@1333456711 after retry RPC@657676800) -> (continuation of Deferred@109596499 after retry RPC@1003943640), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65664a04 -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130) -> (continuation of Deferred@145904798 after retry RPC@160179621) -> (continuation of Deferred@1333456711 after retry RPC@657676800) -> (continuation of Deferred@109596499 after retry RPC@1003943640))) after lookup
18:29:18.804 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:18.806 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.806 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:18.806 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:19.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.009620 21620 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:19.282 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:19.282 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:19.283 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1140), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [2ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [2ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [19ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [20ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [22ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [22ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [39ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [40ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [41ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [41ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [79ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [81ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [82ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [82ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [157ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [159ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [161ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [162ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [162ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [657ms] refreshing cache from master, [657ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [660ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [662ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [664ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [664ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [1136ms] refreshing cache from master, [1137ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [1139ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1554470652(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@66809fc0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130) -> (continuation of Deferred@145904798 after retry RPC@160179621) -> (continuation of Deferred@1333456711 after retry RPC@657676800) -> (continuation of Deferred@109596499 after retry RPC@1003943640) -> (continuation of Deferred@1589121399 after retry RPC@1682291027), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65664a04 -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130) -> (continuation of Deferred@145904798 after retry RPC@160179621) -> (continuation of Deferred@1333456711 after retry RPC@657676800) -> (continuation of Deferred@109596499 after retry RPC@1003943640) -> (continuation of Deferred@1589121399 after retry RPC@1682291027))) after lookup
18:29:19.283 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:19.284 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:19.284 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:19.285 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:19.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.452137 21634 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58924cf836aa406bbedb44253800ef1e)
18:29:19.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.452319 21634 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:19.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.452932 21634 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:19.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:19.456851 21000 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103): Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111)
18:29:19.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.461997 21571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "37ebb4c0266644fa98fa0851dac80a72" is_pre_election: true
18:29:19.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.462469 21571 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d202cde0b06b4bff9c06194f4b40ac5d in term 1.
18:29:19.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.463294 21002 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 58924cf836aa406bbedb44253800ef1e
18:29:19.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.463572 21634 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 2 FOLLOWER]: Leader pre-election won for term 3
18:29:19.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.463686 21634 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58924cf836aa406bbedb44253800ef1e)
18:29:19.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.463754 21634 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 2 FOLLOWER]: Advancing to term 3
18:29:19.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.465692 21634 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:19.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.466292 21634 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 3 election: Requested vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:19.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.466724 21571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "37ebb4c0266644fa98fa0851dac80a72"
18:29:19.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.466924 21571 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 1 FOLLOWER]: Advancing to term 3
18:29:19.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:19.467723 21000 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103): Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111)
18:29:19.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.470691 21571 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d202cde0b06b4bff9c06194f4b40ac5d in term 3.
18:29:19.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.471115 21002 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 58924cf836aa406bbedb44253800ef1e
18:29:19.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.471418 21634 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 3 FOLLOWER]: Leader election won for term 3
18:29:19.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.471822 21634 raft_consensus.cc:697] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 3 LEADER]: Becoming Leader. State: Replica: d202cde0b06b4bff9c06194f4b40ac5d, State: Running, Role: LEADER
18:29:19.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.472304 21634 consensus_queue.cc:237] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:19.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.475701 20816 catalog_manager.cc:5696] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d reported cstate change: term changed from 2 to 3, leader changed from 58924cf836aa406bbedb44253800ef1e (127.20.55.131) to d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129). New cstate: current_term: 3 leader_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } health_report { overall_health: UNKNOWN } } }
18:29:19.542 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:19.542 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:19.543 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1400), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [2ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [2ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [19ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [20ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [22ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [22ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [39ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [40ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [41ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [41ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [79ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [81ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [82ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [82ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [157ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [159ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [161ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [162ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [162ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [657ms] refreshing cache from master, [657ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [660ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [662ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [664ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [664ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [1136ms] refreshing cache from master, [1137ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [1139ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [1141ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [1143ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [1143ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [1397ms] refreshing cache from master, [1397ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [1400ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1554470652(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@66809fc0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130) -> (continuation of Deferred@145904798 after retry RPC@160179621) -> (continuation of Deferred@1333456711 after retry RPC@657676800) -> (continuation of Deferred@109596499 after retry RPC@1003943640) -> (continuation of Deferred@1589121399 after retry RPC@1682291027) -> (continuation of Deferred@339556975 after retry RPC@1576765082), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65664a04 -> wakeup thread Time-limited test -> (continuation of Deferred@1344379272 after retry RPC@121692130) -> (continuation of Deferred@145904798 after retry RPC@160179621) -> (continuation of Deferred@1333456711 after retry RPC@657676800) -> (continuation of Deferred@109596499 after retry RPC@1003943640) -> (continuation of Deferred@1589121399 after retry RPC@1682291027) -> (continuation of Deferred@339556975 after retry RPC@1576765082))) after lookup
18:29:19.544 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:19.545 [DEBUG - kudu-nio-1] (Connection.java:757) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] Successfully connected to peer
18:29:19.560 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xa81c54be, L:/127.0.0.1:42864 - R:/127.20.55.129:43419] using TOKEN/null
18:29:19.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.565505 21571 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 3 FOLLOWER]: Refusing update from remote peer d202cde0b06b4bff9c06194f4b40ac5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
18:29:19.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:19.565615 21000 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d -> Peer 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103): Couldn't send request to peer 58924cf836aa406bbedb44253800ef1e. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:29:19.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.566468 21639 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
18:29:19.604 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.55.131:42103
18:29:19.605 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:29:19.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.606436 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:19.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:19.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:19.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data
18:29:19.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:19.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:19.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:19.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:19.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/logs
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.131:42103
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.131
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40047
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.131
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:19.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:19.936674 21662 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:19.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:19.937017 21662 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:19.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:19.937139 21662 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:19.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:19.945554 21662 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:19.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:19.945778 21662 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.131
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.955994 21662 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.131:42103
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.131
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40047
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.131
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/logs
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:19.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.958163 21662 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:19.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.960059 21662 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:19.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:19.970029 21668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:19.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:19.970916 21667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:19.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:19.973001 21670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:19.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.973196 21662 server_base.cc:1061] running on GCE node
18:29:19.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.973812 21662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:19.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.974829 21662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:19.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.976054 21662 hybrid_clock.cc:648] HybridClock initialized: now 1783535359976005 us; error 39 us; skew 500 ppm
18:29:19.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.976406 21662 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:19.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.978628 21662 webserver.cc:534] Webserver started at http://127.20.55.131:40047/ using document root <none> and password file <none>
18:29:19.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.979497 21662 fs_manager.cc:362] Metadata directory not provided
18:29:19.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.979646 21662 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:19.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.985332 21662 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.000s	sys 0.006s
18:29:19.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.988184 21676 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:19.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.989487 21662 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.989627 21662 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "58924cf836aa406bbedb44253800ef1e"
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:13 on dist-test-slave-4hqz"
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:19.990264 21662 fs_report.cc:389] FS layout report
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/data
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:19.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:19.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:20.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.010316 21662 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:20.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.011188 21662 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:20.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.011427 21662 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:20.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.012296 21662 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.014623 21683 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:29:20.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.020714 21662 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:29:20.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.020810 21662 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
18:29:20.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.020926 21662 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:29:20.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.024039 21662 ts_tablet_manager.cc:616] Registered 1 tablets
18:29:20.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.024121 21662 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.001s
18:29:20.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.024595 21683 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Bootstrap starting.
18:29:20.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.045802 21683 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Log is configured to *not* fsync() on all Append() calls
18:29:20.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.062992 21683 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:29:20.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.063818 21683 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Bootstrap complete.
18:29:20.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.065707 21683 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Time spent bootstrapping tablet: real 0.041s	user 0.026s	sys 0.014s
18:29:20.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.070061 21662 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.131:42103
18:29:20.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.070132 21791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.131:42103 every 8 connection(s)
18:29:20.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.071488 21662 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.071856 21683 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:20.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.072935 21683 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58924cf836aa406bbedb44253800ef1e, State: Initialized, Role: FOLLOWER
18:29:20.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.073904 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 21662
18:29:20.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.074072 21683 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:20.075 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:20.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.079509 21683 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Time spent starting tablet: real 0.014s	user 0.010s	sys 0.003s
18:29:20.096 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:20.097 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.55.129:43419
18:29:20.098 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:29:20.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.098522 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 20985
18:29:20.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.122567 21793 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:20.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.123068 21793 heartbeater.cc:461] Registering TS with master...
18:29:20.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.124295 21793 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:20.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.125538 21108 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:20.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.127509 20743 ts_manager.cc:194] Re-registered known tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:20.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.133046 21792 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:20.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.133157 21792 heartbeater.cc:461] Registering TS with master...
18:29:20.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.133680 21792 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:20.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.134805 20887 ts_manager.cc:194] Re-registered known tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:20.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.138020 21794 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:20.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.138165 21794 heartbeater.cc:461] Registering TS with master...
18:29:20.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.138629 21794 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:20.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.139844 20816 ts_manager.cc:194] Re-registered known tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:20.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.141333 20816 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.131:56345
18:29:20.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:20.232690 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 20985
18:29:20.251 [DEBUG - kudu-nio-1] (Connection.java:239) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] handling channelInactive
18:29:20.252 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state READY due to: connection closed
18:29:20.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:20.252709 21807 negotiation.cc:336] Failed RPC negotiation. Trace:
18:29:20.252 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:20.117246 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.55.129:36005 (local address 127.20.55.131:42103)
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:20.119318 (+  2072us) server_negotiation.cc:207] Beginning negotiation
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:20.119332 (+    14us) server_negotiation.cc:400] Waiting for connection header
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:20.120811 (+  1479us) server_negotiation.cc:408] Connection header received
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:20.120898 (+    87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:20.120905 (+     7us) server_negotiation.cc:462] Received NEGOTIATE request from client
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:20.121153 (+   248us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:20.121314 (+   161us) server_negotiation.cc:227] Negotiated authn=SASL
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:20.122501 (+  1187us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:20.124374 (+  1873us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:20.251194 (+126820us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.55.129:36005: BlockingRecv error: recv got EOF from 127.20.55.129:36005 (error 108)
18:29:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1938,"thread_start_us":3002,"threads_started":1}
18:29:20.253 [DEBUG - Time-limited test] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:20.254 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.255 [DEBUG - kudu-nio-2] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.255 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.261 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:20.261 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:20.261 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=8), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [2ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1653717547(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17d4dccc -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6526513f -> wakeup thread Time-limited test)) after lookup
18:29:20.262 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:20.263 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.263 [DEBUG - kudu-nio-3] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.263 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.281 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:20.282 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:20.282 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [2ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [9ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [10ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [10ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [28ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1653717547(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17d4dccc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@398682004 after retry RPC@1086026746), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6526513f -> wakeup thread Time-limited test -> (continuation of Deferred@398682004 after retry RPC@1086026746))) after lookup
18:29:20.283 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:20.284 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.284 [DEBUG - kudu-nio-4] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.284 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.301 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:20.302 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:20.302 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=49), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [2ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [9ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [10ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [10ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [28ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [29ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [31ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [31ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [48ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1653717547(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17d4dccc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@398682004 after retry RPC@1086026746) -> (continuation of Deferred@1579822192 after retry RPC@1167433964), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6526513f -> wakeup thread Time-limited test -> (continuation of Deferred@398682004 after retry RPC@1086026746) -> (continuation of Deferred@1579822192 after retry RPC@1167433964))) after lookup
18:29:20.303 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:20.304 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.304 [DEBUG - kudu-nio-5] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.304 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.321 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:20.321 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:20.321 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [2ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [9ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [10ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [10ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [28ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [29ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [31ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [31ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [48ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [49ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [51ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [51ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [68ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1653717547(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17d4dccc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@398682004 after retry RPC@1086026746) -> (continuation of Deferred@1579822192 after retry RPC@1167433964) -> (continuation of Deferred@46811367 after retry RPC@1604377753), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6526513f -> wakeup thread Time-limited test -> (continuation of Deferred@398682004 after retry RPC@1086026746) -> (continuation of Deferred@1579822192 after retry RPC@1167433964) -> (continuation of Deferred@46811367 after retry RPC@1604377753))) after lookup
18:29:20.322 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:20.323 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.323 [DEBUG - kudu-nio-6] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.323 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.702 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:20.702 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:20.702 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=449), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [2ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [9ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [10ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [10ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [28ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [29ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [31ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [31ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [48ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [49ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [51ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [51ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [68ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [69ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [70ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [70ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [446ms] refreshing cache from master, [446ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [448ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1653717547(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17d4dccc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@398682004 after retry RPC@1086026746) -> (continuation of Deferred@1579822192 after retry RPC@1167433964) -> (continuation of Deferred@46811367 after retry RPC@1604377753) -> (continuation of Deferred@897756072 after retry RPC@1822424469), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6526513f -> wakeup thread Time-limited test -> (continuation of Deferred@398682004 after retry RPC@1086026746) -> (continuation of Deferred@1579822192 after retry RPC@1167433964) -> (continuation of Deferred@46811367 after retry RPC@1604377753) -> (continuation of Deferred@897756072 after retry RPC@1822424469))) after lookup
18:29:20.703 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:20.704 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.704 [DEBUG - kudu-nio-7] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:20.704 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:21.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.143507 21794 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:21.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.495144 21809 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d202cde0b06b4bff9c06194f4b40ac5d)
18:29:21.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.495353 21809 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:21.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.497339 21809 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:21.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:21.498524 21503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111)
18:29:21.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:21.500196 21503 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419): Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111)
18:29:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.505218 21747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "58924cf836aa406bbedb44253800ef1e" is_pre_election: true
18:29:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.505666 21747 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ebb4c0266644fa98fa0851dac80a72 in term 2.
18:29:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.506604 21502 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, 58924cf836aa406bbedb44253800ef1e; no voters: d202cde0b06b4bff9c06194f4b40ac5d
18:29:21.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.507002 21809 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 3 FOLLOWER]: Leader pre-election won for term 4
18:29:21.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.507133 21809 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d202cde0b06b4bff9c06194f4b40ac5d)
18:29:21.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.507241 21809 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 3 FOLLOWER]: Advancing to term 4
18:29:21.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.509284 21809 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:21.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.509889 21809 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 4 election: Requested vote from peers d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:21.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.510473 21747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "58924cf836aa406bbedb44253800ef1e"
18:29:21.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.510684 21747 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 2 FOLLOWER]: Advancing to term 4
18:29:21.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:21.511348 21503 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419): Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111)
18:29:21.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.514240 21747 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ebb4c0266644fa98fa0851dac80a72 in term 4.
18:29:21.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.514669 21502 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, 58924cf836aa406bbedb44253800ef1e; no voters: d202cde0b06b4bff9c06194f4b40ac5d
18:29:21.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.514966 21809 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 4 FOLLOWER]: Leader election won for term 4
18:29:21.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.515340 21809 raft_consensus.cc:697] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 4 LEADER]: Becoming Leader. State: Replica: 37ebb4c0266644fa98fa0851dac80a72, State: Running, Role: LEADER
18:29:21.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.515828 21809 consensus_queue.cc:237] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:21.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.519419 20816 catalog_manager.cc:5696] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 reported cstate change: term changed from 3 to 4, leader changed from d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129) to 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130). New cstate: current_term: 4 leader_uuid: "37ebb4c0266644fa98fa0851dac80a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } health_report { overall_health: UNKNOWN } } }
18:29:21.681 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:21.681 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:21.681 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1428), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [2ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [9ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [10ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [10ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [28ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [29ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [31ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [31ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [48ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [49ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [51ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [51ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [68ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [69ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [70ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [70ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [446ms] refreshing cache from master, [446ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [448ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [449ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [451ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [451ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [1425ms] refreshing cache from master, [1425ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [1428ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1653717547(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17d4dccc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@398682004 after retry RPC@1086026746) -> (continuation of Deferred@1579822192 after retry RPC@1167433964) -> (continuation of Deferred@46811367 after retry RPC@1604377753) -> (continuation of Deferred@897756072 after retry RPC@1822424469) -> (continuation of Deferred@884273696 after retry RPC@659870326), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6526513f -> wakeup thread Time-limited test -> (continuation of Deferred@398682004 after retry RPC@1086026746) -> (continuation of Deferred@1579822192 after retry RPC@1167433964) -> (continuation of Deferred@46811367 after retry RPC@1604377753) -> (continuation of Deferred@897756072 after retry RPC@1822424469) -> (continuation of Deferred@884273696 after retry RPC@659870326))) after lookup
18:29:21.682 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:21.683 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] Successfully connected to peer
18:29:21.702 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xede92bc5, L:/127.0.0.1:59552 - R:/127.20.55.130:33399] using TOKEN/null
18:29:21.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.708132 21747 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 4 FOLLOWER]: Refusing update from remote peer 37ebb4c0266644fa98fa0851dac80a72: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
18:29:21.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.709172 21814 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
18:29:21.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:21.709630 21503 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 -> Peer d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419): Couldn't send request to peer d202cde0b06b4bff9c06194f4b40ac5d. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:29:21.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.730135 21831 mvcc.cc:204] Tried to move back new op lower bound from 7305360841546252288 to 7305360840772276224. Current Snapshot: MvccSnapshot[applied={T|T < 7305360841546252288}]
18:29:21.746 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.55.129:43419
18:29:21.747 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:21.747906 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/logs
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:21.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.129:43419
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.129
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39253
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.129
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:21.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:22.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:22.091377 21837 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:22.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:22.091658 21837 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:22.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:22.091737 21837 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:22.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:22.100035 21837 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:22.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:22.100204 21837 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.129
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.109931 21837 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.129:43419
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.129
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39253
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:22.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.129
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/logs
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:22.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.111598 21837 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:22.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.113346 21837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:22.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:22.123136 21843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:22.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:22.123426 21842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:22.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:22.123955 21845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:22.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.125080 21837 server_base.cc:1061] running on GCE node
18:29:22.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.125599 21837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:22.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.126547 21837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:22.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.127729 21837 hybrid_clock.cc:648] HybridClock initialized: now 1783535362127663 us; error 63 us; skew 500 ppm
18:29:22.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.128012 21837 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:22.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.130103 21837 webserver.cc:534] Webserver started at http://127.20.55.129:39253/ using document root <none> and password file <none>
18:29:22.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.130825 21837 fs_manager.cc:362] Metadata directory not provided
18:29:22.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.130937 21837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:22.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.136097 21837 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
18:29:22.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.138772 21851 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:22.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.139997 21837 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
18:29:22.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.140126 21837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:22.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d202cde0b06b4bff9c06194f4b40ac5d"
18:29:22.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:22.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.140771 21837 fs_report.cc:389] FS layout report
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/data
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:22.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.160890 21837 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:22.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.161603 21837 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:22.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.161834 21837 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:22.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.162696 21837 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.164664 21858 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:29:22.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.170328 21837 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:29:22.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.170475 21837 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
18:29:22.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.170571 21837 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:29:22.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.173421 21837 ts_tablet_manager.cc:616] Registered 1 tablets
18:29:22.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.173508 21837 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
18:29:22.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.174008 21858 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Bootstrap starting.
18:29:22.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.197086 21858 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Log is configured to *not* fsync() on all Append() calls
18:29:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.216228 21837 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.129:43419
18:29:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.216328 21966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.129:43419 every 8 connection(s)
18:29:22.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.217747 21837 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:22.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.221694 21858 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:29:22.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.222515 21858 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Bootstrap complete.
18:29:22.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.224213 21858 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Time spent bootstrapping tablet: real 0.051s	user 0.032s	sys 0.015s
18:29:22.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.227228 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 21837
18:29:22.227 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:22.241 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:22.249 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.55.130:33399
18:29:22.250 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:29:22.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.250401 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 21487
18:29:22.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.254343 21858 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:22.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.258788 21858 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: d202cde0b06b4bff9c06194f4b40ac5d, State: Initialized, Role: FOLLOWER
18:29:22.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.259712 21858 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:22.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.262247 21858 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Time spent starting tablet: real 0.038s	user 0.011s	sys 0.002s
18:29:22.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.301221 21612 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.308029 21968 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.308138 21967 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:22.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.308508 21967 heartbeater.cc:461] Registering TS with master...
18:29:22.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.308513 21968 heartbeater.cc:461] Registering TS with master...
18:29:22.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.309689 21967 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:22.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.309722 21968 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:22.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.311968 21969 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:22.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.312088 21969 heartbeater.cc:461] Registering TS with master...
18:29:22.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.312602 21969 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:22.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.312846 20743 ts_manager.cc:194] Re-registered known tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:22.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.313853 20816 ts_manager.cc:194] Re-registered known tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:22.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.314028 20886 ts_manager.cc:194] Re-registered known tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:22.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.315439 20816 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.129:33607
18:29:22.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.318854 21969 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:22.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:22.412165 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 21487
18:29:22.432 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] handling channelInactive
18:29:22.432 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:22.432 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state READY due to: connection closed
18:29:22.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:22.432529 21981 negotiation.cc:336] Failed RPC negotiation. Trace:
18:29:22.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:22.298809 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.55.130:58547 (local address 127.20.55.129:43419)
18:29:22.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:22.306248 (+  7439us) server_negotiation.cc:207] Beginning negotiation
18:29:22.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:22.306257 (+     9us) server_negotiation.cc:400] Waiting for connection header
18:29:22.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:22.306290 (+    33us) server_negotiation.cc:408] Connection header received
18:29:22.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:22.306411 (+   121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
18:29:22.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:22.306418 (+     7us) server_negotiation.cc:462] Received NEGOTIATE request from client
18:29:22.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:22.306695 (+   277us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
18:29:22.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:22.307050 (+   355us) server_negotiation.cc:227] Negotiated authn=SASL
18:29:22.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:22.412909 (+105859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
18:29:22.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:22.414850 (+  1941us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
18:29:22.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:22.431046 (+ 16196us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.55.130:58547: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:29:22.433 [DEBUG - Time-limited test] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:22.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":7297,"thread_start_us":7165,"threads_started":1}
18:29:22.434 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.434 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.434 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.441 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:22.441 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:22.441 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@266670298(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@70c18e88 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad7243f -> wakeup thread Time-limited test)) after lookup
18:29:22.441 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:22.442 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.442 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.443 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.461 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:22.461 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:22.462 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [9ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [11ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [11ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [29ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@266670298(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@70c18e88 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@530249519 after retry RPC@218095167), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad7243f -> wakeup thread Time-limited test -> (continuation of Deferred@530249519 after retry RPC@218095167))) after lookup
18:29:22.462 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:22.463 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.463 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.463 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.481 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:22.481 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:22.481 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=49), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [9ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [11ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [11ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [29ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [30ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [31ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [31ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [49ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@266670298(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@70c18e88 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@530249519 after retry RPC@218095167) -> (continuation of Deferred@1183715222 after retry RPC@888287070), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad7243f -> wakeup thread Time-limited test -> (continuation of Deferred@530249519 after retry RPC@218095167) -> (continuation of Deferred@1183715222 after retry RPC@888287070))) after lookup
18:29:22.482 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:22.483 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.483 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.483 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.581 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:22.581 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:22.582 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=149), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [9ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [11ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [11ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [29ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [30ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [31ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [31ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [49ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [50ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [51ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [51ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [149ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@266670298(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@70c18e88 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@530249519 after retry RPC@218095167) -> (continuation of Deferred@1183715222 after retry RPC@888287070) -> (continuation of Deferred@523149412 after retry RPC@973337600), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad7243f -> wakeup thread Time-limited test -> (continuation of Deferred@530249519 after retry RPC@218095167) -> (continuation of Deferred@1183715222 after retry RPC@888287070) -> (continuation of Deferred@523149412 after retry RPC@973337600))) after lookup
18:29:22.582 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:22.583 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.584 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:22.584 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:23.022 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:23.022 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:23.023 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=590), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [9ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [11ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [11ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [29ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [30ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [31ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [31ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [49ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [50ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [51ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [51ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [149ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [150ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [152ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [152ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [587ms] refreshing cache from master, [587ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [589ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@266670298(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@70c18e88 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@530249519 after retry RPC@218095167) -> (continuation of Deferred@1183715222 after retry RPC@888287070) -> (continuation of Deferred@523149412 after retry RPC@973337600) -> (continuation of Deferred@2134173512 after retry RPC@1513271130), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad7243f -> wakeup thread Time-limited test -> (continuation of Deferred@530249519 after retry RPC@218095167) -> (continuation of Deferred@1183715222 after retry RPC@888287070) -> (continuation of Deferred@523149412 after retry RPC@973337600) -> (continuation of Deferred@2134173512 after retry RPC@1513271130))) after lookup
18:29:23.023 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:23.024 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:23.024 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:23.025 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:23.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.493383 21983 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:29:23.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.493640 21983 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:23.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.495426 21983 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:23.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:23.496750 21854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:23.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:23.498400 21854 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:23.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.502239 21747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "58924cf836aa406bbedb44253800ef1e" is_pre_election: true
18:29:23.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.503065 21852 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d202cde0b06b4bff9c06194f4b40ac5d; no voters: 37ebb4c0266644fa98fa0851dac80a72, 58924cf836aa406bbedb44253800ef1e
18:29:23.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.503420 21983 raft_consensus.cc:2749] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
18:29:23.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.657232 21988 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 37ebb4c0266644fa98fa0851dac80a72)
18:29:23.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.657486 21988 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:23.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.659264 21988 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:23.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:23.660792 21679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:23.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:23.662474 21679 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.667793 21922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "58924cf836aa406bbedb44253800ef1e" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" is_pre_election: true
18:29:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.668175 21922 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58924cf836aa406bbedb44253800ef1e in term 3.
18:29:23.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.669040 21678 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58924cf836aa406bbedb44253800ef1e, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 37ebb4c0266644fa98fa0851dac80a72
18:29:23.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.669412 21988 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 4 FOLLOWER]: Leader pre-election won for term 5
18:29:23.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.669544 21988 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 37ebb4c0266644fa98fa0851dac80a72)
18:29:23.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.669649 21988 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 4 FOLLOWER]: Advancing to term 5
18:29:23.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.671934 21988 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:23.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.672595 21988 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 5 election: Requested vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:23.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.673166 21922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "58924cf836aa406bbedb44253800ef1e" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d"
18:29:23.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.673403 21922 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 3 FOLLOWER]: Advancing to term 5
18:29:23.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:23.673919 21679 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:23.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.677138 21922 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58924cf836aa406bbedb44253800ef1e in term 5.
18:29:23.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.677608 21678 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58924cf836aa406bbedb44253800ef1e, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 37ebb4c0266644fa98fa0851dac80a72
18:29:23.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.677950 21988 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 5 FOLLOWER]: Leader election won for term 5
18:29:23.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.678323 21988 raft_consensus.cc:697] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 5 LEADER]: Becoming Leader. State: Replica: 58924cf836aa406bbedb44253800ef1e, State: Running, Role: LEADER
18:29:23.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.678879 21988 consensus_queue.cc:237] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:23.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.682572 20816 catalog_manager.cc:5696] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e reported cstate change: term changed from 4 to 5, leader changed from 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130) to 58924cf836aa406bbedb44253800ef1e (127.20.55.131). New cstate: current_term: 5 leader_uuid: "58924cf836aa406bbedb44253800ef1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } health_report { overall_health: HEALTHY } } }
18:29:23.762 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:23.762 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:23.762 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1330), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [8ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [9ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [11ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [11ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [29ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [30ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [31ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [31ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [49ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [50ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [51ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [51ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [149ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [150ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [152ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [152ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [587ms] refreshing cache from master, [587ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [589ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [591ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [593ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [593ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [1327ms] refreshing cache from master, [1327ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [1329ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@266670298(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@70c18e88 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@530249519 after retry RPC@218095167) -> (continuation of Deferred@1183715222 after retry RPC@888287070) -> (continuation of Deferred@523149412 after retry RPC@973337600) -> (continuation of Deferred@2134173512 after retry RPC@1513271130) -> (continuation of Deferred@1116767444 after retry RPC@219655075), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad7243f -> wakeup thread Time-limited test -> (continuation of Deferred@530249519 after retry RPC@218095167) -> (continuation of Deferred@1183715222 after retry RPC@888287070) -> (continuation of Deferred@523149412 after retry RPC@973337600) -> (continuation of Deferred@2134173512 after retry RPC@1513271130) -> (continuation of Deferred@1116767444 after retry RPC@219655075))) after lookup
18:29:23.763 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:23.764 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] Successfully connected to peer
18:29:23.796 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0x803ed302, L:/127.0.0.1:40506 - R:/127.20.55.131:42103] using TOKEN/null
18:29:23.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.802174 21922 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 5 FOLLOWER]: Refusing update from remote peer 58924cf836aa406bbedb44253800ef1e: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
18:29:23.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:23.802682 21679 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e -> Peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Couldn't send request to peer 37ebb4c0266644fa98fa0851dac80a72. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:29:23.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.803107 21988 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
18:29:23.842 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.55.130:33399
18:29:23.842 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:29:23.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:23.843464 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:23.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:23.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:23.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data
18:29:23.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:23.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:23.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/logs
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.130:33399
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.130
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42109
18:29:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.130
18:29:23.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:23.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:23.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:23.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:24.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:24.176780 22014 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:24.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:24.177140 22014 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:24.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:24.177264 22014 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:24.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:24.185806 22014 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:24.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:24.186036 22014 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.130
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.196803 22014 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.130:33399
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.130
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42109
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:24.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.130
18:29:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/logs
18:29:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:24.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.198922 22014 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:24.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.200802 22014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:24.210469 22020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:24.210548 22019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:24.211563 22022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:24.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.212410 22014 server_base.cc:1061] running on GCE node
18:29:24.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.213013 22014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:24.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.213958 22014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:24.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.215188 22014 hybrid_clock.cc:648] HybridClock initialized: now 1783535364215138 us; error 33 us; skew 500 ppm
18:29:24.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.215531 22014 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:24.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.217785 22014 webserver.cc:534] Webserver started at http://127.20.55.130:42109/ using document root <none> and password file <none>
18:29:24.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.218622 22014 fs_manager.cc:362] Metadata directory not provided
18:29:24.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.218750 22014 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:24.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.224412 22014 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
18:29:24.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.227350 22028 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:24.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.228771 22014 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
18:29:24.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.228958 22014 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:24.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "37ebb4c0266644fa98fa0851dac80a72"
18:29:24.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:24.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.229746 22014 fs_report.cc:389] FS layout report
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/data
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:24.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.241777 22014 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:24.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.242697 22014 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:24.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.243021 22014 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.244005 22014 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:24.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.246143 22035 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:29:24.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.251806 22014 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:29:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.251910 22014 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
18:29:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.252043 22014 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:29:24.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.255633 22014 ts_tablet_manager.cc:616] Registered 1 tablets
18:29:24.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.255750 22014 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.003s	sys 0.000s
18:29:24.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.256256 22035 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap starting.
18:29:24.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.281754 22035 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Log is configured to *not* fsync() on all Append() calls
18:29:24.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.298877 22014 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.130:33399
18:29:24.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.298936 22145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.130:33399 every 8 connection(s)
18:29:24.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.300658 22014 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:24.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.300956 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 22014
18:29:24.301 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:24.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.318377 22035 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:29:24.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.319229 22035 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap complete.
18:29:24.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.321216 22035 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Time spent bootstrapping tablet: real 0.065s	user 0.043s	sys 0.012s
18:29:24.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.329100 22035 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.330866 22035 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ebb4c0266644fa98fa0851dac80a72, State: Initialized, Role: FOLLOWER
18:29:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.331588 22035 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:24.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.336040 22035 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Time spent starting tablet: real 0.015s	user 0.011s	sys 0.005s
18:29:24.340 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:24.341 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.55.131:42103
18:29:24.341 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:29:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.342172 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 21662
18:29:24.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.344386 22147 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:24.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.344486 22146 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:24.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.344794 22147 heartbeater.cc:461] Registering TS with master...
18:29:24.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.344797 22146 heartbeater.cc:461] Registering TS with master...
18:29:24.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.345961 22146 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:24.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.346700 22147 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:24.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.349846 20743 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:24.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.351107 20886 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:24.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.359143 22148 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:24.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.359287 22148 heartbeater.cc:461] Registering TS with master...
18:29:24.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.359771 22148 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:24.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.360976 20816 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:24.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.362777 20816 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.130:53087
18:29:24.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.365522 22148 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:24.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.366849 21787 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:24.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:24.468729 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 21662
18:29:24.488 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:24.488 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] handling channelInactive
18:29:24.488 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state READY due to: connection closed
18:29:24.489 [DEBUG - Time-limited test] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:24.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:24.489704 22163 negotiation.cc:336] Failed RPC negotiation. Trace:
18:29:24.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:24.466743 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.55.131:45507 (local address 127.20.55.130:33399)
18:29:24.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:24.467524 (+   781us) server_negotiation.cc:207] Beginning negotiation
18:29:24.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:24.467534 (+    10us) server_negotiation.cc:400] Waiting for connection header
18:29:24.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:24.488213 (+ 20679us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.55.131:45507: BlockingRecv error: recv got EOF from 127.20.55.131:45507 (error 108)
18:29:24.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":439,"threads_started":1}
18:29:24.490 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.490 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.490 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.501 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:24.502 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:24.502 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [1ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [1ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@603725734(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1306f2c9 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4915b242 -> wakeup thread Time-limited test)) after lookup
18:29:24.502 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:24.503 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.503 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.503 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.521 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:24.521 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:24.521 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [1ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [1ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [13ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [14ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [14ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [31ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@603725734(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1306f2c9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1005113119 after retry RPC@422458073), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4915b242 -> wakeup thread Time-limited test -> (continuation of Deferred@1005113119 after retry RPC@422458073))) after lookup
18:29:24.522 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:24.523 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.523 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.523 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.561 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:24.561 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:24.562 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [1ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [1ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [13ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [14ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [14ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [31ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [33ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [34ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [34ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [72ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@603725734(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1306f2c9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1005113119 after retry RPC@422458073) -> (continuation of Deferred@433962574 after retry RPC@1896738646), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4915b242 -> wakeup thread Time-limited test -> (continuation of Deferred@1005113119 after retry RPC@422458073) -> (continuation of Deferred@433962574 after retry RPC@1896738646))) after lookup
18:29:24.562 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:24.563 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.563 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.564 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.602 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:24.602 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:24.602 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=113), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [1ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [1ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [13ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [14ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [14ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [31ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [33ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [34ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [34ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [72ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [73ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [74ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [75ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [112ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@603725734(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1306f2c9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1005113119 after retry RPC@422458073) -> (continuation of Deferred@433962574 after retry RPC@1896738646) -> (continuation of Deferred@1371119244 after retry RPC@1330691250), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4915b242 -> wakeup thread Time-limited test -> (continuation of Deferred@1005113119 after retry RPC@422458073) -> (continuation of Deferred@433962574 after retry RPC@1896738646) -> (continuation of Deferred@1371119244 after retry RPC@1330691250))) after lookup
18:29:24.603 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:24.604 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.604 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.604 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.762 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:24.762 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:24.762 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=274), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [1ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [1ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [13ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [14ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [14ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [31ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [33ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [34ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [34ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [72ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [73ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [74ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [75ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [112ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [114ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [115ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [116ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [270ms] refreshing cache from master, [270ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [272ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@603725734(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1306f2c9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1005113119 after retry RPC@422458073) -> (continuation of Deferred@433962574 after retry RPC@1896738646) -> (continuation of Deferred@1371119244 after retry RPC@1330691250) -> (continuation of Deferred@1792137999 after retry RPC@539905995), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4915b242 -> wakeup thread Time-limited test -> (continuation of Deferred@1005113119 after retry RPC@422458073) -> (continuation of Deferred@433962574 after retry RPC@1896738646) -> (continuation of Deferred@1371119244 after retry RPC@1330691250) -> (continuation of Deferred@1792137999 after retry RPC@539905995))) after lookup
18:29:24.763 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:24.764 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.765 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:24.765 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:25.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.659665 22165 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:29:25.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.659997 22165 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:25.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.661763 22165 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:25.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:25.667369 22029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111)
18:29:25.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:25.668936 22029 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103): Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111)
18:29:25.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.672499 21922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" is_pre_election: true
18:29:25.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.673233 22030 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72; no voters: 58924cf836aa406bbedb44253800ef1e, d202cde0b06b4bff9c06194f4b40ac5d
18:29:25.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.673602 22165 raft_consensus.cc:2749] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
18:29:25.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.861272 22170 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58924cf836aa406bbedb44253800ef1e)
18:29:25.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.861452 22170 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:25.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.862094 22170 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:25.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:25.864411 21852 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103): Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111)
18:29:25.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.869861 22100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "37ebb4c0266644fa98fa0851dac80a72" is_pre_election: true
18:29:25.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.870220 22100 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d202cde0b06b4bff9c06194f4b40ac5d in term 4.
18:29:25.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.871083 21854 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 58924cf836aa406bbedb44253800ef1e
18:29:25.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.871346 22170 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 5 FOLLOWER]: Leader pre-election won for term 6
18:29:25.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.871467 22170 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58924cf836aa406bbedb44253800ef1e)
18:29:25.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.871549 22170 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 5 FOLLOWER]: Advancing to term 6
18:29:25.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.873505 22170 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:25.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.874119 22170 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 6 election: Requested vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:25.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.874547 22100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "37ebb4c0266644fa98fa0851dac80a72"
18:29:25.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.874735 22100 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 4 FOLLOWER]: Advancing to term 6
18:29:25.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:25.875638 21852 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103): Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111)
18:29:25.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.878502 22100 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d202cde0b06b4bff9c06194f4b40ac5d in term 6.
18:29:25.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.878916 21854 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 58924cf836aa406bbedb44253800ef1e
18:29:25.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.879176 22170 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 6 FOLLOWER]: Leader election won for term 6
18:29:25.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.879616 22170 raft_consensus.cc:697] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 6 LEADER]: Becoming Leader. State: Replica: d202cde0b06b4bff9c06194f4b40ac5d, State: Running, Role: LEADER
18:29:25.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.880081 22170 consensus_queue.cc:237] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:25.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:25.883987 20816 catalog_manager.cc:5696] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d reported cstate change: term changed from 5 to 6, leader changed from 58924cf836aa406bbedb44253800ef1e (127.20.55.131) to d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129). New cstate: current_term: 6 leader_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } health_report { overall_health: UNKNOWN } } }
18:29:26.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:26.339097 21852 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d -> Peer 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103): Couldn't send request to peer 58924cf836aa406bbedb44253800ef1e. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:29:26.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:26.384516 22100 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 6 FOLLOWER]: Refusing update from remote peer d202cde0b06b4bff9c06194f4b40ac5d: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
18:29:26.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:26.385409 22170 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
18:29:26.742 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:26.742 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:26.742 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2253), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [1ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [1ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [13ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [14ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [14ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [31ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [33ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [34ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [34ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [72ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [73ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [74ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [75ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [112ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [114ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [115ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [116ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [270ms] refreshing cache from master, [270ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [272ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [274ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [276ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [276ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [2250ms] refreshing cache from master, [2250ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [2252ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@603725734(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1306f2c9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1005113119 after retry RPC@422458073) -> (continuation of Deferred@433962574 after retry RPC@1896738646) -> (continuation of Deferred@1371119244 after retry RPC@1330691250) -> (continuation of Deferred@1792137999 after retry RPC@539905995) -> (continuation of Deferred@1011980301 after retry RPC@1850984539), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4915b242 -> wakeup thread Time-limited test -> (continuation of Deferred@1005113119 after retry RPC@422458073) -> (continuation of Deferred@433962574 after retry RPC@1896738646) -> (continuation of Deferred@1371119244 after retry RPC@1330691250) -> (continuation of Deferred@1792137999 after retry RPC@539905995) -> (continuation of Deferred@1011980301 after retry RPC@1850984539))) after lookup
18:29:26.743 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:26.744 [DEBUG - kudu-nio-6] (Connection.java:757) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] Successfully connected to peer
18:29:26.771 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0xca090e6f, L:/127.0.0.1:53352 - R:/127.20.55.129:43419] using TOKEN/null
18:29:26.798 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.55.131:42103
18:29:26.800 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:29:26.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:26.800678 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/logs
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.131:42103
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.131
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40047
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.131
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:27.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:27.125510 22199 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:27.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:27.125939 22199 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:27.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:27.126036 22199 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:27.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:27.134172 22199 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:27.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:27.134323 22199 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.131
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.144413 22199 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.131:42103
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.131
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40047
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.131
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/logs
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:27.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.146092 22199 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:27.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.147959 22199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:27.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:27.157537 22204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:27.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:27.157831 22205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:27.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:27.158790 22207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:27.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.159479 22199 server_base.cc:1061] running on GCE node
18:29:27.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.160145 22199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:27.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.161164 22199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:27.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.162451 22199 hybrid_clock.cc:648] HybridClock initialized: now 1783535367162390 us; error 42 us; skew 500 ppm
18:29:27.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.162807 22199 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:27.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.164997 22199 webserver.cc:534] Webserver started at http://127.20.55.131:40047/ using document root <none> and password file <none>
18:29:27.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.165761 22199 fs_manager.cc:362] Metadata directory not provided
18:29:27.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.165901 22199 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:27.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.171277 22199 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
18:29:27.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.174033 22213 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:27.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.175391 22199 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.175568 22199 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "58924cf836aa406bbedb44253800ef1e"
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:13 on dist-test-slave-4hqz"
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.176301 22199 fs_report.cc:389] FS layout report
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/data
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:27.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:27.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:27.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:27.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:27.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:27.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:27.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.195042 22199 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:27.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.195921 22199 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:27.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.196249 22199 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:27.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.197254 22199 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:27.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.199553 22220 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:29:27.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.205207 22199 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:29:27.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.205317 22199 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.000s	sys 0.001s
18:29:27.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.205456 22199 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:29:27.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.208410 22199 ts_tablet_manager.cc:616] Registered 1 tablets
18:29:27.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.208531 22199 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.001s
18:29:27.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.208850 22220 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Bootstrap starting.
18:29:27.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.234333 22220 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Log is configured to *not* fsync() on all Append() calls
18:29:27.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.259037 22199 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.131:42103
18:29:27.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.259085 22328 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.131:42103 every 8 connection(s)
18:29:27.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.260475 22199 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:27.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.264145 22220 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:29:27.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.264921 22220 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Bootstrap complete.
18:29:27.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.266551 22220 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Time spent bootstrapping tablet: real 0.058s	user 0.041s	sys 0.016s
18:29:27.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.269224 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 22199
18:29:27.270 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:27.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.274979 22220 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:27.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.275945 22220 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58924cf836aa406bbedb44253800ef1e, State: Initialized, Role: FOLLOWER
18:29:27.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.276708 22220 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:27.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.286973 22220 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Time spent starting tablet: real 0.020s	user 0.017s	sys 0.000s
18:29:27.300 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:27.301 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.55.129:43419
18:29:27.301 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:29:27.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.301594 22330 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:27.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.301940 22330 heartbeater.cc:461] Registering TS with master...
18:29:27.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.302276 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 21837
18:29:27.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.302973 22330 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:27.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.306850 22329 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:27.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.306982 22329 heartbeater.cc:461] Registering TS with master...
18:29:27.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.307420 22329 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:27.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.307492 20744 ts_manager.cc:194] Re-registered known tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:27.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.313457 20886 ts_manager.cc:194] Re-registered known tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:27.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.314707 22331 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.314848 22331 heartbeater.cc:461] Registering TS with master...
18:29:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.315352 22331 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:27.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.316522 20815 ts_manager.cc:194] Re-registered known tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:27.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.318166 20815 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.131:46063
18:29:27.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.320538 22331 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.325788 21962 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:27.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:27.424863 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 21837
18:29:27.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:27.444257 20730 connection.cc:573] server connection from 127.20.55.129:48867 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:29:27.444 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:27.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:27.444343 20800 connection.cc:573] server connection from 127.20.55.129:33607 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:29:27.444 [DEBUG - kudu-nio-6] (Connection.java:239) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] handling channelInactive
18:29:27.445 [DEBUG - kudu-nio-6] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state READY due to: connection closed
18:29:27.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:27.445026 20872 connection.cc:573] server connection from 127.20.55.129:46963 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:29:27.445 [DEBUG - Time-limited test] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:27.446 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.446 [DEBUG - kudu-nio-7] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.446 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.461 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:27.461 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:27.461 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=16), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1811071263(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1b93cd2a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@234f45d3 -> wakeup thread Time-limited test)) after lookup
18:29:27.462 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:27.463 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.463 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.463 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.482 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:27.483 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:27.483 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [17ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [18ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1811071263(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1b93cd2a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@234f45d3 -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217))) after lookup
18:29:27.483 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:27.485 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.485 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.485 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.502 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:27.502 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:27.503 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=58), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [17ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [18ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [38ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [40ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [40ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [53ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [57ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1811071263(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1b93cd2a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217) -> (continuation of Deferred@2082298432 after retry RPC@182980907), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@234f45d3 -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217) -> (continuation of Deferred@2082298432 after retry RPC@182980907))) after lookup
18:29:27.503 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:27.505 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.505 [DEBUG - kudu-nio-2] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.505 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.603 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:27.603 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:27.603 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=158), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [17ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [18ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [38ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [40ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [40ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [53ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [57ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [58ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [60ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [60ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [154ms] refreshing cache from master, [154ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [157ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1811071263(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1b93cd2a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217) -> (continuation of Deferred@2082298432 after retry RPC@182980907) -> (continuation of Deferred@81223583 after retry RPC@1316779879), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@234f45d3 -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217) -> (continuation of Deferred@2082298432 after retry RPC@182980907) -> (continuation of Deferred@81223583 after retry RPC@1316779879))) after lookup
18:29:27.604 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:27.605 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.605 [DEBUG - kudu-nio-3] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:27.605 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:28.081 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:28.082 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:28.082 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=637), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [17ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [18ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [38ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [40ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [40ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [53ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [57ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [58ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [60ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [60ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [154ms] refreshing cache from master, [154ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [157ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [159ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [160ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [160ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [633ms] refreshing cache from master, [634ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [636ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1811071263(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1b93cd2a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217) -> (continuation of Deferred@2082298432 after retry RPC@182980907) -> (continuation of Deferred@81223583 after retry RPC@1316779879) -> (continuation of Deferred@1135651236 after retry RPC@1851406842), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@234f45d3 -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217) -> (continuation of Deferred@2082298432 after retry RPC@182980907) -> (continuation of Deferred@81223583 after retry RPC@1316779879) -> (continuation of Deferred@1135651236 after retry RPC@1851406842))) after lookup
18:29:28.083 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:28.084 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:28.084 [DEBUG - kudu-nio-4] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:28.085 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:28.662 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:28.662 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:28.662 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1217), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [17ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [18ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [38ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [40ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [40ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [53ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [57ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [58ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [60ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [60ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [154ms] refreshing cache from master, [154ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [157ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [159ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [160ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [160ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [633ms] refreshing cache from master, [634ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [636ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [638ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [639ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [640ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [1213ms] refreshing cache from master, [1214ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [1216ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1811071263(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1b93cd2a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217) -> (continuation of Deferred@2082298432 after retry RPC@182980907) -> (continuation of Deferred@81223583 after retry RPC@1316779879) -> (continuation of Deferred@1135651236 after retry RPC@1851406842) -> (continuation of Deferred@768877006 after retry RPC@310270055), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@234f45d3 -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217) -> (continuation of Deferred@2082298432 after retry RPC@182980907) -> (continuation of Deferred@81223583 after retry RPC@1316779879) -> (continuation of Deferred@1135651236 after retry RPC@1851406842) -> (continuation of Deferred@768877006 after retry RPC@310270055))) after lookup
18:29:28.663 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:28.664 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:28.664 [DEBUG - kudu-nio-5] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:28.664 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:28.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.925654 22343 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d202cde0b06b4bff9c06194f4b40ac5d)
18:29:28.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.925915 22343 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:28.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.926788 22343 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:28.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:28.937395 22030 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419): Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111)
18:29:28.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.943706 22284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "58924cf836aa406bbedb44253800ef1e" is_pre_election: true
18:29:28.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.944118 22284 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ebb4c0266644fa98fa0851dac80a72 in term 5.
18:29:28.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.944983 22029 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, 58924cf836aa406bbedb44253800ef1e; no voters: d202cde0b06b4bff9c06194f4b40ac5d
18:29:28.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.945284 22343 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 6 FOLLOWER]: Leader pre-election won for term 7
18:29:28.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.945429 22343 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d202cde0b06b4bff9c06194f4b40ac5d)
18:29:28.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.945528 22343 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 6 FOLLOWER]: Advancing to term 7
18:29:28.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.947686 22343 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:28.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.948388 22343 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 7 election: Requested vote from peers d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:28.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.948902 22284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "58924cf836aa406bbedb44253800ef1e"
18:29:28.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.949129 22284 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 5 FOLLOWER]: Advancing to term 7
18:29:28.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:28.950429 22030 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419): Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111)
18:29:28.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.953190 22284 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ebb4c0266644fa98fa0851dac80a72 in term 7.
18:29:28.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.953621 22029 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, 58924cf836aa406bbedb44253800ef1e; no voters: d202cde0b06b4bff9c06194f4b40ac5d
18:29:28.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.953962 22343 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 7 FOLLOWER]: Leader election won for term 7
18:29:28.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.954418 22343 raft_consensus.cc:697] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 7 LEADER]: Becoming Leader. State: Replica: 37ebb4c0266644fa98fa0851dac80a72, State: Running, Role: LEADER
18:29:28.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.954974 22343 consensus_queue.cc:237] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:28.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:28.958740 20815 catalog_manager.cc:5696] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 reported cstate change: term changed from 6 to 7, leader changed from d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129) to 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130). New cstate: current_term: 7 leader_uuid: "37ebb4c0266644fa98fa0851dac80a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } health_report { overall_health: UNKNOWN } } }
18:29:29.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:29.374956 22030 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 -> Peer d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419): Couldn't send request to peer d202cde0b06b4bff9c06194f4b40ac5d. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:29:29.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:29.492638 22284 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 7 FOLLOWER]: Refusing update from remote peer 37ebb4c0266644fa98fa0851dac80a72: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
18:29:29.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:29.493552 22343 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
18:29:30.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:30.876324 22030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111) [suppressed 5 similar messages]
18:29:32.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:32.107120 22030 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 -> Peer d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419): Couldn't send request to peer d202cde0b06b4bff9c06194f4b40ac5d. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
18:29:32.162 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:32.162 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:32.163 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=4718), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [2ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [17ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [18ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [38ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [40ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [40ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [53ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [57ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [58ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [60ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [60ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [154ms] refreshing cache from master, [154ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [157ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [159ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [160ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [160ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [633ms] refreshing cache from master, [634ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [636ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [638ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [639ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [640ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [1213ms] refreshing cache from master, [1214ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [1216ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [1218ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1219ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [1219ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [4713ms] refreshing cache from master, [4714ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [4716ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1811071263(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@1b93cd2a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217) -> (continuation of Deferred@2082298432 after retry RPC@182980907) -> (continuation of Deferred@81223583 after retry RPC@1316779879) -> (continuation of Deferred@1135651236 after retry RPC@1851406842) -> (continuation of Deferred@768877006 after retry RPC@310270055) -> (continuation of Deferred@1318157306 after retry RPC@785170223), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@234f45d3 -> wakeup thread Time-limited test -> (continuation of Deferred@503193320 after retry RPC@1153872217) -> (continuation of Deferred@2082298432 after retry RPC@182980907) -> (continuation of Deferred@81223583 after retry RPC@1316779879) -> (continuation of Deferred@1135651236 after retry RPC@1851406842) -> (continuation of Deferred@768877006 after retry RPC@310270055) -> (continuation of Deferred@1318157306 after retry RPC@785170223))) after lookup
18:29:32.163 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:32.164 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] Successfully connected to peer
18:29:32.182 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0xebe13d54, L:/127.0.0.1:57824 - R:/127.20.55.130:33399] using TOKEN/null
18:29:32.204 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.55.129:43419
18:29:32.205 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:29:32.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.205675 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:32.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:32.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:32.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data
18:29:32.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/logs
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.129:43419
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.129
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39253
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.129
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:32.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:32.537909 22384 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:32.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:32.538183 22384 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:32.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:32.538266 22384 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:32.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:32.546480 22384 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:32.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:32.546628 22384 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.129
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.556160 22384 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.129:43419
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.129
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39253
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.129
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/logs
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:32.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:32.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:32.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:32.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:32.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:32.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:32.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:32.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.557906 22384 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:32.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.559748 22384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:32.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:32.569319 22389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:32.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:32.569394 22392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:32.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:32.569285 22390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:32.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.570406 22384 server_base.cc:1061] running on GCE node
18:29:32.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.571062 22384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:32.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.571923 22384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.573113 22384 hybrid_clock.cc:648] HybridClock initialized: now 1783535372573062 us; error 47 us; skew 500 ppm
18:29:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.573376 22384 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:32.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.575513 22384 webserver.cc:534] Webserver started at http://127.20.55.129:39253/ using document root <none> and password file <none>
18:29:32.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.576231 22384 fs_manager.cc:362] Metadata directory not provided
18:29:32.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.576323 22384 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:32.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.581518 22384 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
18:29:32.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.584622 22398 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:32.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.586073 22384 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.586207 22384 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d202cde0b06b4bff9c06194f4b40ac5d"
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.586999 22384 fs_report.cc:389] FS layout report
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/data
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:32.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:32.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.605145 22384 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:32.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.605917 22384 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:32.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.606153 22384 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:32.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.607120 22384 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:32.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.609177 22405 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:29:32.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.615069 22384 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:29:32.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.615152 22384 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
18:29:32.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.615254 22384 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:29:32.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.618203 22384 ts_tablet_manager.cc:616] Registered 1 tablets
18:29:32.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.618296 22384 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
18:29:32.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.618788 22405 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Bootstrap starting.
18:29:32.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.643307 22405 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Log is configured to *not* fsync() on all Append() calls
18:29:32.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.663110 22384 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.129:43419
18:29:32.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.663160 22513 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.129:43419 every 8 connection(s)
18:29:32.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.664606 22384 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:32.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.672619 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 22384
18:29:32.673 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:32.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.687963 22405 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:29:32.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.688694 22405 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Bootstrap complete.
18:29:32.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.693110 22405 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Time spent bootstrapping tablet: real 0.075s	user 0.041s	sys 0.020s
18:29:32.697 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:32.697 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.55.130:33399
18:29:32.698 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:29:32.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.698537 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 22014
18:29:32.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.709174 22514 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:32.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.709587 22514 heartbeater.cc:461] Registering TS with master...
18:29:32.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.710925 22515 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:32.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.711104 22515 heartbeater.cc:461] Registering TS with master...
18:29:32.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.711971 22405 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:32.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.713078 22405 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: d202cde0b06b4bff9c06194f4b40ac5d, State: Initialized, Role: FOLLOWER
18:29:32.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.713413 22514 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:32.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.714113 22405 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:32.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.714673 22515 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:32.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.717466 20744 ts_manager.cc:194] Re-registered known tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:32.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.718729 20887 ts_manager.cc:194] Re-registered known tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:32.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.721513 22141 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:32.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.721884 22516 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:32.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.722043 22516 heartbeater.cc:461] Registering TS with master...
18:29:32.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.722586 22516 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:32.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.723961 20816 ts_manager.cc:194] Re-registered known tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:32.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.725471 20816 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.129:42567
18:29:32.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.725883 22405 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Time spent starting tablet: real 0.032s	user 0.008s	sys 0.005s
18:29:32.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.728420 22516 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:32.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:32.821593 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 22014
18:29:32.846 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] handling channelInactive
18:29:32.846 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:32.846 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state READY due to: connection closed
18:29:32.847 [DEBUG - Time-limited test] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:32.848 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.848 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.848 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.861 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:32.861 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:32.861 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [15ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@787899648(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@65888236 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4762d755 -> wakeup thread Time-limited test)) after lookup
18:29:32.862 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:32.863 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.863 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.863 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.881 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:32.881 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:32.881 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [15ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [16ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [17ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [17ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [34ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@787899648(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@65888236 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4762d755 -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448))) after lookup
18:29:32.881 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:32.882 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.882 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.882 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.901 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:32.901 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:32.901 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=55), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [15ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [16ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [17ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [17ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [34ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [35ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [36ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [36ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [55ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@787899648(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@65888236 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448) -> (continuation of Deferred@1624602116 after retry RPC@1766401994), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4762d755 -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448) -> (continuation of Deferred@1624602116 after retry RPC@1766401994))) after lookup
18:29:32.902 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:32.903 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.903 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.903 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.981 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:32.982 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:32.982 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=135), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [15ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [16ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [17ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [17ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [34ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [35ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [36ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [36ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [55ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [56ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [57ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [57ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [135ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@787899648(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@65888236 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448) -> (continuation of Deferred@1624602116 after retry RPC@1766401994) -> (continuation of Deferred@1685186419 after retry RPC@1143401280), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4762d755 -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448) -> (continuation of Deferred@1624602116 after retry RPC@1766401994) -> (continuation of Deferred@1685186419 after retry RPC@1143401280))) after lookup
18:29:32.983 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:32.984 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.984 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:32.984 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:33.122 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:33.122 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:33.122 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=276), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [15ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [16ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [17ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [17ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [34ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [35ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [36ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [36ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [55ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [56ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [57ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [57ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [135ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [136ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [138ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [138ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [272ms] refreshing cache from master, [273ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [275ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@787899648(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@65888236 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448) -> (continuation of Deferred@1624602116 after retry RPC@1766401994) -> (continuation of Deferred@1685186419 after retry RPC@1143401280) -> (continuation of Deferred@272476516 after retry RPC@1664236744), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4762d755 -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448) -> (continuation of Deferred@1624602116 after retry RPC@1766401994) -> (continuation of Deferred@1685186419 after retry RPC@1143401280) -> (continuation of Deferred@272476516 after retry RPC@1664236744))) after lookup
18:29:33.123 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:33.124 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:33.124 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:33.124 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:33.742 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:33.742 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:33.743 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=896), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [15ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [16ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [17ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [17ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [34ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [35ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [36ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [36ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [55ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [56ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [57ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [57ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [135ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [136ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [138ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [138ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [272ms] refreshing cache from master, [273ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [275ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [277ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [278ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [278ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [893ms] refreshing cache from master, [893ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [896ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@787899648(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@65888236 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448) -> (continuation of Deferred@1624602116 after retry RPC@1766401994) -> (continuation of Deferred@1685186419 after retry RPC@1143401280) -> (continuation of Deferred@272476516 after retry RPC@1664236744) -> (continuation of Deferred@1304621373 after retry RPC@1071927345), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4762d755 -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448) -> (continuation of Deferred@1624602116 after retry RPC@1766401994) -> (continuation of Deferred@1685186419 after retry RPC@1143401280) -> (continuation of Deferred@272476516 after retry RPC@1664236744) -> (continuation of Deferred@1304621373 after retry RPC@1071927345))) after lookup
18:29:33.743 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:33.744 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:33.745 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:33.745 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:34.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.043766 22529 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:29:34.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.044122 22529 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:34.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.045964 22529 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:34.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:34.057482 22401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:34.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:34.059520 22401 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:34.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.063566 22284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "58924cf836aa406bbedb44253800ef1e" is_pre_election: true
18:29:34.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.064574 22399 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d202cde0b06b4bff9c06194f4b40ac5d; no voters: 37ebb4c0266644fa98fa0851dac80a72, 58924cf836aa406bbedb44253800ef1e
18:29:34.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.065047 22529 raft_consensus.cc:2749] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
18:29:34.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.173055 22534 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 37ebb4c0266644fa98fa0851dac80a72)
18:29:34.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.173249 22534 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:34.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.174944 22534 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:34.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:34.176512 22216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:34.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:34.178283 22216 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:34.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.184159 22469 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "58924cf836aa406bbedb44253800ef1e" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" is_pre_election: true
18:29:34.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.184643 22469 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58924cf836aa406bbedb44253800ef1e in term 6.
18:29:34.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.185617 22215 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58924cf836aa406bbedb44253800ef1e, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 37ebb4c0266644fa98fa0851dac80a72
18:29:34.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.185984 22534 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 7 FOLLOWER]: Leader pre-election won for term 8
18:29:34.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.186110 22534 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 37ebb4c0266644fa98fa0851dac80a72)
18:29:34.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.186215 22534 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 7 FOLLOWER]: Advancing to term 8
18:29:34.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.188301 22534 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:34.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.188876 22534 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 8 election: Requested vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:34.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.189435 22469 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "58924cf836aa406bbedb44253800ef1e" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d"
18:29:34.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.189646 22469 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 6 FOLLOWER]: Advancing to term 8
18:29:34.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:34.190147 22216 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:34.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.193394 22469 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58924cf836aa406bbedb44253800ef1e in term 8.
18:29:34.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.193781 22215 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58924cf836aa406bbedb44253800ef1e, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 37ebb4c0266644fa98fa0851dac80a72
18:29:34.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.194125 22534 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 8 FOLLOWER]: Leader election won for term 8
18:29:34.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.194592 22534 raft_consensus.cc:697] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 8 LEADER]: Becoming Leader. State: Replica: 58924cf836aa406bbedb44253800ef1e, State: Running, Role: LEADER
18:29:34.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.195115 22534 consensus_queue.cc:237] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:34.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.198824 20816 catalog_manager.cc:5696] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e reported cstate change: term changed from 7 to 8, leader changed from 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130) to 58924cf836aa406bbedb44253800ef1e (127.20.55.131). New cstate: current_term: 8 leader_uuid: "58924cf836aa406bbedb44253800ef1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } health_report { overall_health: HEALTHY } } }
18:29:34.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:34.586894 22216 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e -> Peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Couldn't send request to peer 37ebb4c0266644fa98fa0851dac80a72. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:29:34.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.632457 22469 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 8 FOLLOWER]: Refusing update from remote peer 58924cf836aa406bbedb44253800ef1e: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
18:29:34.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:34.633302 22534 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
18:29:37.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:37.130195 22216 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e -> Peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Couldn't send request to peer 37ebb4c0266644fa98fa0851dac80a72. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
18:29:37.622 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:37.622 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:37.623 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=4776), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [15ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [16ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [17ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [17ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [34ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [35ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [36ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [36ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [55ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [56ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [57ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [57ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [135ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [136ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [138ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [138ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [272ms] refreshing cache from master, [273ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [275ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [277ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [278ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [278ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [893ms] refreshing cache from master, [893ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [896ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [897ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [899ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [899ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [4773ms] refreshing cache from master, [4773ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [4775ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@787899648(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@65888236 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448) -> (continuation of Deferred@1624602116 after retry RPC@1766401994) -> (continuation of Deferred@1685186419 after retry RPC@1143401280) -> (continuation of Deferred@272476516 after retry RPC@1664236744) -> (continuation of Deferred@1304621373 after retry RPC@1071927345) -> (continuation of Deferred@620420679 after retry RPC@956458801), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4762d755 -> wakeup thread Time-limited test -> (continuation of Deferred@2089432483 after retry RPC@755207448) -> (continuation of Deferred@1624602116 after retry RPC@1766401994) -> (continuation of Deferred@1685186419 after retry RPC@1143401280) -> (continuation of Deferred@272476516 after retry RPC@1664236744) -> (continuation of Deferred@1304621373 after retry RPC@1071927345) -> (continuation of Deferred@620420679 after retry RPC@956458801))) after lookup
18:29:37.623 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:37.624 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] Successfully connected to peer
18:29:37.642 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0xd27d51c5, L:/127.0.0.1:44072 - R:/127.20.55.131:42103] using TOKEN/null
18:29:37.680 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.55.130:33399
18:29:37.681 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:37.681807 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/logs
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.130:33399
18:29:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.130
18:29:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42109
18:29:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.130
18:29:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:38.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:38.011883 22580 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:38.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:38.012164 22580 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:38.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:38.012245 22580 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:38.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:38.020318 22580 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:38.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:38.020474 22580 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.130
18:29:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.030308 22580 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data
18:29:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.130:33399
18:29:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.130
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42109
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.130
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/logs
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:38.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:38.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.032012 22580 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:38.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.033757 22580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:38.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:38.043488 22586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:38.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:38.043550 22585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:38.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:38.044314 22588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:38.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.044649 22580 server_base.cc:1061] running on GCE node
18:29:38.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.045281 22580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:38.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.046319 22580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:38.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.047573 22580 hybrid_clock.cc:648] HybridClock initialized: now 1783535378047521 us; error 26 us; skew 500 ppm
18:29:38.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.048035 22580 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:38.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.050455 22580 webserver.cc:534] Webserver started at http://127.20.55.130:42109/ using document root <none> and password file <none>
18:29:38.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.051177 22580 fs_manager.cc:362] Metadata directory not provided
18:29:38.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.051265 22580 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:38.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.056594 22580 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
18:29:38.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.059654 22594 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:38.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.061064 22580 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
18:29:38.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.061208 22580 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:38.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "37ebb4c0266644fa98fa0851dac80a72"
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.061895 22580 fs_report.cc:389] FS layout report
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/data
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:38.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.104939 22580 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:38.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.106127 22580 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:38.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.106495 22580 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:38.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.107774 22580 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:38.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.110651 22601 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:29:38.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.116771 22580 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:29:38.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.116895 22580 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
18:29:38.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.117038 22580 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:29:38.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.121758 22580 ts_tablet_manager.cc:616] Registered 1 tablets
18:29:38.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.121883 22580 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.000s	sys 0.002s
18:29:38.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.122413 22601 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap starting.
18:29:38.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.145421 22601 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Log is configured to *not* fsync() on all Append() calls
18:29:38.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.184052 22601 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:29:38.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.184850 22601 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap complete.
18:29:38.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.186092 22601 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Time spent bootstrapping tablet: real 0.064s	user 0.028s	sys 0.034s
18:29:38.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.192912 22601 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:38.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.193774 22601 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ebb4c0266644fa98fa0851dac80a72, State: Initialized, Role: FOLLOWER
18:29:38.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.194591 22601 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:38.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.196216 22601 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.003s
18:29:38.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.199512 22580 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.130:33399
18:29:38.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.199555 22710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.130:33399 every 8 connection(s)
18:29:38.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.201288 22580 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:38.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.210464 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 22580
18:29:38.211 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:38.238 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:38.239 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.55.131:42103
18:29:38.239 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:29:38.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.239778 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 22199
18:29:38.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.245410 22711 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:38.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.245748 22711 heartbeater.cc:461] Registering TS with master...
18:29:38.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.246963 22711 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:38.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.249657 22713 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:38.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.249790 22713 heartbeater.cc:461] Registering TS with master...
18:29:38.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.250233 22713 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:38.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.251745 20886 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:38.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.251793 20815 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:38.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.253387 20815 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.130:51579
18:29:38.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.253603 22712 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:38.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.253732 22712 heartbeater.cc:461] Registering TS with master...
18:29:38.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.254268 22712 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:38.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.257835 20743 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:38.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.261484 22324 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:38.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:38.365159 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 22199
18:29:38.386 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] handling channelInactive
18:29:38.387 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state READY due to: connection closed
18:29:38.387 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:38.388 [DEBUG - Time-limited test] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:38.390 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.390 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.390 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.401 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:38.402 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:38.402 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [3ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [3ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@162729135(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@2322d947 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1191e711 -> wakeup thread Time-limited test)) after lookup
18:29:38.403 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:38.404 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.404 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.404 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.421 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:38.421 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:38.422 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [3ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [3ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [15ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [17ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [17ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [34ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@162729135(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@2322d947 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@27077588 after retry RPC@522907033), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1191e711 -> wakeup thread Time-limited test -> (continuation of Deferred@27077588 after retry RPC@522907033))) after lookup
18:29:38.422 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:38.423 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.424 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.424 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.462 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:38.462 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:38.463 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=75), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [3ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [3ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [15ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [17ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [17ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [34ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [35ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [37ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [37ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [75ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@162729135(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@2322d947 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@27077588 after retry RPC@522907033) -> (continuation of Deferred@382276745 after retry RPC@909822321), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1191e711 -> wakeup thread Time-limited test -> (continuation of Deferred@27077588 after retry RPC@522907033) -> (continuation of Deferred@382276745 after retry RPC@909822321))) after lookup
18:29:38.463 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:38.464 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.464 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.464 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.581 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:38.581 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:38.582 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=194), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [3ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [3ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [15ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [17ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [17ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [34ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [35ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [37ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [37ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [75ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [76ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [77ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [77ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [194ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@162729135(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@2322d947 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@27077588 after retry RPC@522907033) -> (continuation of Deferred@382276745 after retry RPC@909822321) -> (continuation of Deferred@1400566729 after retry RPC@871090582), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1191e711 -> wakeup thread Time-limited test -> (continuation of Deferred@27077588 after retry RPC@522907033) -> (continuation of Deferred@382276745 after retry RPC@909822321) -> (continuation of Deferred@1400566729 after retry RPC@871090582))) after lookup
18:29:38.582 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:38.583 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.583 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:38.583 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:39.062 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:39.062 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:39.062 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=675), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [3ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [3ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [15ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [17ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [17ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [34ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [35ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [37ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [37ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [75ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [76ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [77ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [77ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [194ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [195ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [196ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [196ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [672ms] refreshing cache from master, [672ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [675ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@162729135(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@2322d947 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@27077588 after retry RPC@522907033) -> (continuation of Deferred@382276745 after retry RPC@909822321) -> (continuation of Deferred@1400566729 after retry RPC@871090582) -> (continuation of Deferred@1551002969 after retry RPC@825796664), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1191e711 -> wakeup thread Time-limited test -> (continuation of Deferred@27077588 after retry RPC@522907033) -> (continuation of Deferred@382276745 after retry RPC@909822321) -> (continuation of Deferred@1400566729 after retry RPC@871090582) -> (continuation of Deferred@1551002969 after retry RPC@825796664))) after lookup
18:29:39.063 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:39.064 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:39.064 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state CONNECTING due to: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:39.064 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103
18:29:39.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.256901 22713 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:39.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.500344 22725 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:29:39.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.500648 22725 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:39.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.502413 22725 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:39.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:39.507638 22595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111)
18:29:39.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:39.509313 22595 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103): Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111)
18:29:39.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.513509 22469 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" is_pre_election: true
18:29:39.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.514315 22596 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72; no voters: 58924cf836aa406bbedb44253800ef1e, d202cde0b06b4bff9c06194f4b40ac5d
18:29:39.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.514751 22725 raft_consensus.cc:2749] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
18:29:39.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.688905 22730 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58924cf836aa406bbedb44253800ef1e)
18:29:39.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.689152 22730 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:39.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.689939 22730 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:39.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:39.691370 22399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111)
18:29:39.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:39.692973 22399 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103): Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111)
18:29:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.698141 22664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "37ebb4c0266644fa98fa0851dac80a72" is_pre_election: true
18:29:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.698614 22664 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d202cde0b06b4bff9c06194f4b40ac5d in term 7.
18:29:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.699561 22401 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 58924cf836aa406bbedb44253800ef1e
18:29:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.699868 22730 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 8 FOLLOWER]: Leader pre-election won for term 9
18:29:39.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.699994 22730 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 58924cf836aa406bbedb44253800ef1e)
18:29:39.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.700078 22730 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 8 FOLLOWER]: Advancing to term 9
18:29:39.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.702123 22730 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:39.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.702785 22730 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 9 election: Requested vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:39.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.703176 22664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "37ebb4c0266644fa98fa0851dac80a72"
18:29:39.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.703365 22664 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 7 FOLLOWER]: Advancing to term 9
18:29:39.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:39.704151 22399 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103): Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111)
18:29:39.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.707989 22664 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d202cde0b06b4bff9c06194f4b40ac5d in term 9.
18:29:39.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.708453 22401 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 58924cf836aa406bbedb44253800ef1e
18:29:39.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.708834 22730 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 9 FOLLOWER]: Leader election won for term 9
18:29:39.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.709259 22730 raft_consensus.cc:697] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 9 LEADER]: Becoming Leader. State: Replica: d202cde0b06b4bff9c06194f4b40ac5d, State: Running, Role: LEADER
18:29:39.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.709664 22730 consensus_queue.cc:237] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:39.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:39.713588 20815 catalog_manager.cc:5696] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d reported cstate change: term changed from 8 to 9, leader changed from 58924cf836aa406bbedb44253800ef1e (127.20.55.131) to d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129). New cstate: current_term: 9 leader_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } health_report { overall_health: UNKNOWN } } }
18:29:40.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:40.284835 22399 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d -> Peer 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103): Couldn't send request to peer 58924cf836aa406bbedb44253800ef1e. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.131:42103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:29:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:40.319382 22664 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 9 FOLLOWER]: Refusing update from remote peer d202cde0b06b4bff9c06194f4b40ac5d: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
18:29:40.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:40.320407 22748 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
18:29:40.781 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:40.782 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:40.782 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2394), Traces: [0ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [3ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [3ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [15ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [17ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [17ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [34ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [35ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [37ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [37ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [75ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [76ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [77ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [77ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [194ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [195ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [196ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [196ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [672ms] refreshing cache from master, [672ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [675ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [676ms] sending RPC to server 58924cf836aa406bbedb44253800ef1e, [677ms] received response from server 58924cf836aa406bbedb44253800ef1e: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [677ms] delaying RPC due to: Network error: Failed to connect to peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103): Connection refused: /127.20.55.131:42103, [2391ms] refreshing cache from master, [2392ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [2394ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@162729135(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@2322d947 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@27077588 after retry RPC@522907033) -> (continuation of Deferred@382276745 after retry RPC@909822321) -> (continuation of Deferred@1400566729 after retry RPC@871090582) -> (continuation of Deferred@1551002969 after retry RPC@825796664) -> (continuation of Deferred@73872563 after retry RPC@1894397295), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1191e711 -> wakeup thread Time-limited test -> (continuation of Deferred@27077588 after retry RPC@522907033) -> (continuation of Deferred@382276745 after retry RPC@909822321) -> (continuation of Deferred@1400566729 after retry RPC@871090582) -> (continuation of Deferred@1551002969 after retry RPC@825796664) -> (continuation of Deferred@73872563 after retry RPC@1894397295))) after lookup
18:29:40.782 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:40.783 [DEBUG - kudu-nio-5] (Connection.java:757) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] Successfully connected to peer
18:29:40.810 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x3f2a3151, L:/127.0.0.1:38928 - R:/127.20.55.129:43419] using TOKEN/null
18:29:40.844 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.55.131:42103
18:29:40.844 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:40.844995 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/logs
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:40.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.131:42103
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.131
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40047
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.131
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:40.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:41.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:41.169977 22764 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:41.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:41.170269 22764 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:41.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:41.170364 22764 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:41.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:41.178341 22764 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:41.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:41.178519 22764 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.131
18:29:41.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.188318 22764 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:41.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:41.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:41.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:41.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:41.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data
18:29:41.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:41.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:41.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:41.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.131:42103
18:29:41.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.131
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40047
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.131
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/logs
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:41.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:41.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.190258 22764 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:41.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.192190 22764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:41.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:41.201431 22770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:41.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:41.201468 22769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:41.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:41.202766 22772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:41.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.203517 22764 server_base.cc:1061] running on GCE node
18:29:41.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.204144 22764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:41.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.205133 22764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:41.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.206347 22764 hybrid_clock.cc:648] HybridClock initialized: now 1783535381206294 us; error 36 us; skew 500 ppm
18:29:41.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.206703 22764 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:41.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.208673 22764 webserver.cc:534] Webserver started at http://127.20.55.131:40047/ using document root <none> and password file <none>
18:29:41.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.209419 22764 fs_manager.cc:362] Metadata directory not provided
18:29:41.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.209554 22764 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:41.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.214855 22764 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
18:29:41.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.217440 22778 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:41.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.218703 22764 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
18:29:41.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.218914 22764 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:41.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "58924cf836aa406bbedb44253800ef1e"
18:29:41.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:13 on dist-test-slave-4hqz"
18:29:41.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.219599 22764 fs_report.cc:389] FS layout report
18:29:41.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:41.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/wal
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/data
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:41.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.268123 22764 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:41.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.269053 22764 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:41.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.269454 22764 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:41.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.270503 22764 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:41.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.272589 22786 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:29:41.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.278666 22764 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:29:41.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.278815 22764 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
18:29:41.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.278966 22764 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:29:41.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.282232 22764 ts_tablet_manager.cc:616] Registered 1 tablets
18:29:41.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.282371 22764 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
18:29:41.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.282811 22786 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Bootstrap starting.
18:29:41.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.313496 22786 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Log is configured to *not* fsync() on all Append() calls
18:29:41.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.322131 22764 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.131:42103
18:29:41.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.322206 22894 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.131:42103 every 8 connection(s)
18:29:41.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.323714 22764 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-2/data/info.pb
18:29:41.333 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:41.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.332616 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 22764
18:29:41.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.350437 22895 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:41.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.350778 22895 heartbeater.cc:461] Registering TS with master...
18:29:41.351 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:41.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.351913 22895 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:41.352 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.55.129:43419
18:29:41.352 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:29:41.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.352756 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 22384
18:29:41.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.355751 20886 ts_manager.cc:194] Re-registered known tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:41.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.357378 22897 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:41.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.357547 22897 heartbeater.cc:461] Registering TS with master...
18:29:41.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.358098 22897 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:41.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.359953 20815 ts_manager.cc:194] Re-registered known tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:41.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.361774 20815 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.131:55861
18:29:41.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.367026 22786 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:29:41.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.367731 22786 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Bootstrap complete.
18:29:41.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.369920 22786 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Time spent bootstrapping tablet: real 0.087s	user 0.052s	sys 0.022s
18:29:41.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.373101 22896 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:41.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.373282 22896 heartbeater.cc:461] Registering TS with master...
18:29:41.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.373631 22896 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:41.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.375126 20743 ts_manager.cc:194] Re-registered known tserver with Master: 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:41.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.375978 22786 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:41.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.376909 22786 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58924cf836aa406bbedb44253800ef1e, State: Initialized, Role: FOLLOWER
18:29:41.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.377719 22786 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:41.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.379168 22897 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:41.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.383368 22786 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.004s
18:29:41.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.386080 22848 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 8 FOLLOWER]: Advancing to term 9
18:29:41.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.393386 22509 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:41.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:41.500341 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 22384
18:29:41.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:41.518929 22596 connection.cc:573] server connection from 127.20.55.129:59959 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:29:41.519 [DEBUG - kudu-nio-5] (Connection.java:239) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] handling channelInactive
18:29:41.519 [DEBUG - kudu-nio-5] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state READY due to: connection closed
18:29:41.519 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:41.520 [DEBUG - Time-limited test] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:41.521 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.521 [DEBUG - kudu-nio-6] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.521 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.541 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:41.541 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:41.541 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [1ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [20ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@294343876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@30eb4ad7 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10672ed -> wakeup thread Time-limited test)) after lookup
18:29:41.542 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:41.543 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.543 [DEBUG - kudu-nio-7] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.543 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.561 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:41.561 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:41.562 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [1ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [20ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [21ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [23ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [23ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [41ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@294343876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@30eb4ad7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@758412406 after retry RPC@521665063), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10672ed -> wakeup thread Time-limited test -> (continuation of Deferred@758412406 after retry RPC@521665063))) after lookup
18:29:41.562 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:41.563 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.563 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.563 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.581 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:41.581 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:41.582 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=61), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [1ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [20ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [21ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [23ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [23ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [41ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [42ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [43ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [43ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [61ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@294343876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@30eb4ad7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@758412406 after retry RPC@521665063) -> (continuation of Deferred@1758519003 after retry RPC@1211549476), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10672ed -> wakeup thread Time-limited test -> (continuation of Deferred@758412406 after retry RPC@521665063) -> (continuation of Deferred@1758519003 after retry RPC@1211549476))) after lookup
18:29:41.582 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:41.583 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.583 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.583 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.642 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:41.642 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:41.643 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=123), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [1ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [20ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [21ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [23ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [23ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [41ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [42ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [43ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [43ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [61ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [62ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [63ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [64ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [122ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@294343876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@30eb4ad7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@758412406 after retry RPC@521665063) -> (continuation of Deferred@1758519003 after retry RPC@1211549476) -> (continuation of Deferred@1022928878 after retry RPC@1598112334), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10672ed -> wakeup thread Time-limited test -> (continuation of Deferred@758412406 after retry RPC@521665063) -> (continuation of Deferred@1758519003 after retry RPC@1211549476) -> (continuation of Deferred@1022928878 after retry RPC@1598112334))) after lookup
18:29:41.643 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:41.644 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.644 [DEBUG - kudu-nio-2] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.644 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.982 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:41.982 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)[L]]}]
18:29:41.982 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=462), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [1ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [20ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [21ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [23ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [23ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [41ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [42ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [43ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [43ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [61ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [62ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [63ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [64ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [122ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [123ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [124ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [125ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [458ms] refreshing cache from master, [459ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [461ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@294343876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@30eb4ad7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@758412406 after retry RPC@521665063) -> (continuation of Deferred@1758519003 after retry RPC@1211549476) -> (continuation of Deferred@1022928878 after retry RPC@1598112334) -> (continuation of Deferred@1464105775 after retry RPC@37879995), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10672ed -> wakeup thread Time-limited test -> (continuation of Deferred@758412406 after retry RPC@521665063) -> (continuation of Deferred@1758519003 after retry RPC@1211549476) -> (continuation of Deferred@1022928878 after retry RPC@1598112334) -> (continuation of Deferred@1464105775 after retry RPC@37879995))) after lookup
18:29:41.983 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] connecting to peer
18:29:41.984 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.984 [DEBUG - kudu-nio-3] (Connection.java:709) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] cleaning up while in state CONNECTING due to: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:41.984 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419
18:29:42.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:42.894285 22919 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d202cde0b06b4bff9c06194f4b40ac5d)
18:29:42.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:42.894553 22919 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:42.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:42.897456 22919 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:42.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:42.897889 22780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111)
18:29:42.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:42.899745 22780 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419): Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111)
18:29:42.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:42.903508 22664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "58924cf836aa406bbedb44253800ef1e" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "37ebb4c0266644fa98fa0851dac80a72" is_pre_election: true
18:29:42.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:42.904289 22781 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58924cf836aa406bbedb44253800ef1e; no voters: 37ebb4c0266644fa98fa0851dac80a72, d202cde0b06b4bff9c06194f4b40ac5d
18:29:42.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:42.904718 22919 raft_consensus.cc:2749] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
18:29:43.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.002125 22924 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d202cde0b06b4bff9c06194f4b40ac5d)
18:29:43.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.002413 22924 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:43.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.003261 22924 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:43.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.005925 22596 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419): Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111)
18:29:43.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.009876 22848 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "58924cf836aa406bbedb44253800ef1e" is_pre_election: true
18:29:43.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.010195 22848 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ebb4c0266644fa98fa0851dac80a72 in term 9.
18:29:43.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.010746 22595 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, 58924cf836aa406bbedb44253800ef1e; no voters: d202cde0b06b4bff9c06194f4b40ac5d
18:29:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.011021 22924 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 9 FOLLOWER]: Leader pre-election won for term 10
18:29:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.011147 22924 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d202cde0b06b4bff9c06194f4b40ac5d)
18:29:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.011215 22924 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 9 FOLLOWER]: Advancing to term 10
18:29:43.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.013255 22924 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:43.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.013808 22924 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 10 election: Requested vote from peers d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419), 58924cf836aa406bbedb44253800ef1e (127.20.55.131:42103)
18:29:43.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.014266 22848 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "37ebb4c0266644fa98fa0851dac80a72" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "58924cf836aa406bbedb44253800ef1e"
18:29:43.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.014508 22848 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 9 FOLLOWER]: Advancing to term 10
18:29:43.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.015298 22596 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419): Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111)
18:29:43.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.016788 22848 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ebb4c0266644fa98fa0851dac80a72 in term 10.
18:29:43.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.017169 22595 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ebb4c0266644fa98fa0851dac80a72, 58924cf836aa406bbedb44253800ef1e; no voters: d202cde0b06b4bff9c06194f4b40ac5d
18:29:43.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.017616 22924 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 10 FOLLOWER]: Leader election won for term 10
18:29:43.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.018196 22924 raft_consensus.cc:697] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 10 LEADER]: Becoming Leader. State: Replica: 37ebb4c0266644fa98fa0851dac80a72, State: Running, Role: LEADER
18:29:43.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.018777 22924 consensus_queue.cc:237] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:43.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.022665 20815 catalog_manager.cc:5696] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 reported cstate change: term changed from 9 to 10, leader changed from d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129) to 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130). New cstate: current_term: 10 leader_uuid: "37ebb4c0266644fa98fa0851dac80a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } health_report { overall_health: UNKNOWN } } }
18:29:43.161 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:43.161 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:43.162 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1641), Traces: [0ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [1ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [1ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [20ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [21ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [23ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [23ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [41ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [42ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [43ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [43ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [61ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [62ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [63ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [64ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [122ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [123ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [124ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [125ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [458ms] refreshing cache from master, [459ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [461ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [463ms] sending RPC to server d202cde0b06b4bff9c06194f4b40ac5d, [464ms] received response from server d202cde0b06b4bff9c06194f4b40ac5d: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [464ms] delaying RPC due to: Network error: Failed to connect to peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419): Connection refused: /127.20.55.129:43419, [1638ms] refreshing cache from master, [1639ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [1641ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@294343876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@30eb4ad7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@758412406 after retry RPC@521665063) -> (continuation of Deferred@1758519003 after retry RPC@1211549476) -> (continuation of Deferred@1022928878 after retry RPC@1598112334) -> (continuation of Deferred@1464105775 after retry RPC@37879995) -> (continuation of Deferred@1337265215 after retry RPC@1566608042), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10672ed -> wakeup thread Time-limited test -> (continuation of Deferred@758412406 after retry RPC@521665063) -> (continuation of Deferred@1758519003 after retry RPC@1211549476) -> (continuation of Deferred@1022928878 after retry RPC@1598112334) -> (continuation of Deferred@1464105775 after retry RPC@37879995) -> (continuation of Deferred@1337265215 after retry RPC@1566608042))) after lookup
18:29:43.162 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:43.163 [DEBUG - kudu-nio-4] (Connection.java:757) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] Successfully connected to peer
18:29:43.195 [DEBUG - kudu-nio-4] (Negotiator.java:919) Authenticated connection [id: 0x65ee6a5c, L:/127.0.0.1:49574 - R:/127.20.55.130:33399] using TOKEN/null
18:29:43.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.201354 22848 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 10 FOLLOWER]: Refusing update from remote peer 37ebb4c0266644fa98fa0851dac80a72: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 10 index: 38. (index mismatch)
18:29:43.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.202219 22924 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
18:29:43.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.204566 22596 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 -> Peer d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419): Couldn't send request to peer d202cde0b06b4bff9c06194f4b40ac5d. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.129:43419: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:29:43.230 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.55.129:43419
18:29:43.230 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:29:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.231536 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data
18:29:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/logs
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.129:43419
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.129
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39253
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.129
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:43.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.562819 22948 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:43.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.563105 22948 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:43.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.563187 22948 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:43.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.572094 22948 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:43.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.572261 22948 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.129
18:29:43.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.582614 22948 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:43.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.129:43419
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.129
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39253
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.129
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/logs
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:43.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:43.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.584174 22948 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:43.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.586037 22948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:43.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.596124 22957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:43.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.596093 22955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:43.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.597229 22954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:43.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.597513 22948 server_base.cc:1061] running on GCE node
18:29:43.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.598098 22948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:43.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.599102 22948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:43.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.600306 22948 hybrid_clock.cc:648] HybridClock initialized: now 1783535383600237 us; error 61 us; skew 500 ppm
18:29:43.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.600567 22948 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:43.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.603217 22948 webserver.cc:534] Webserver started at http://127.20.55.129:39253/ using document root <none> and password file <none>
18:29:43.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.603957 22948 fs_manager.cc:362] Metadata directory not provided
18:29:43.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.604046 22948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:43.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.609104 22948 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.004s
18:29:43.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.611724 22963 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:43.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.613008 22948 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
18:29:43.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.613135 22948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:43.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d202cde0b06b4bff9c06194f4b40ac5d"
18:29:43.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:43.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.613755 22948 fs_report.cc:389] FS layout report
18:29:43.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:43.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/wal
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/data
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:43.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.633466 22948 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:43.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.634279 22948 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:43.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.634590 22948 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:43.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.635471 22948 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:43.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.637568 22970 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:29:43.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.643498 22948 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:29:43.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.643587 22948 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
18:29:43.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.643687 22948 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:29:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.646626 22948 ts_tablet_manager.cc:616] Registered 1 tablets
18:29:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.646713 22948 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
18:29:43.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.647218 22970 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Bootstrap starting.
18:29:43.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.674553 22970 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Log is configured to *not* fsync() on all Append() calls
18:29:43.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.695268 22948 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.129:43419
18:29:43.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.695331 23079 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.129:43419 every 8 connection(s)
18:29:43.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.696666 22948 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-0/data/info.pb
18:29:43.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.698643 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 22948
18:29:43.699 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:43.742 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:43.742 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.55.130:33399
18:29:43.743 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:29:43.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.743100 23080 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:43.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.743244 22970 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:29:43.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.743533 23080 heartbeater.cc:461] Registering TS with master...
18:29:43.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.743975 22970 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Bootstrap complete.
18:29:43.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.744763 23080 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:43.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.745076 23082 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:43.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.745217 23082 heartbeater.cc:461] Registering TS with master...
18:29:43.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.745549 22970 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Time spent bootstrapping tablet: real 0.099s	user 0.063s	sys 0.009s
18:29:43.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.745818 23082 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:43.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.746531 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 22580
18:29:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.750378 20887 ts_manager.cc:194] Re-registered known tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:43.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.751704 20815 ts_manager.cc:194] Re-registered known tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:43.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.753794 20815 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.129:49069
18:29:43.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.753098 22970 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:43.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.754046 22970 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: d202cde0b06b4bff9c06194f4b40ac5d, State: Initialized, Role: FOLLOWER
18:29:43.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.755067 22970 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:43.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.758637 23081 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:43.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.758798 23081 heartbeater.cc:461] Registering TS with master...
18:29:43.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.759213 23081 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:43.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.760978 20743 ts_manager.cc:194] Re-registered known tserver with Master: d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:43.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.766083 22970 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.014s
18:29:43.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.766650 23082 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:43.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.780879 22705 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:43.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:43.879881 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 22580
18:29:43.899 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:43.899 [DEBUG - kudu-nio-4] (Connection.java:239) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] handling channelInactive
18:29:43.899 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state READY due to: connection closed
18:29:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:43.899838 23096 negotiation.cc:336] Failed RPC negotiation. Trace:
18:29:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:43.770264 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.55.130:33729 (local address 127.20.55.129:43419)
18:29:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:43.780880 (+ 10616us) server_negotiation.cc:207] Beginning negotiation
18:29:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:43.780889 (+     9us) server_negotiation.cc:400] Waiting for connection header
18:29:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:43.780924 (+    35us) server_negotiation.cc:408] Connection header received
18:29:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:43.781025 (+   101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
18:29:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:43.781036 (+    11us) server_negotiation.cc:462] Received NEGOTIATE request from client
18:29:43.900 [DEBUG - Time-limited test] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:43.781401 (+   365us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
18:29:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:43.781670 (+   269us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
18:29:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 18:29:43.898249 (+116579us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.55.130:33729: BlockingRecv error: recv got EOF from 127.20.55.130:33729 (error 108)
18:29:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":10462,"thread_start_us":10311,"threads_started":1}
18:29:43.901 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:43.901 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:43.901 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:43.922 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:43.922 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:43.922 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [22ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1321678960(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17abced8 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4eb36e7 -> wakeup thread Time-limited test)) after lookup
18:29:43.923 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:43.924 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:43.924 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:43.924 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:43.941 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:43.941 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:43.941 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [22ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [24ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [25ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [25ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [41ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1321678960(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17abced8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@857602730 after retry RPC@1847373344), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4eb36e7 -> wakeup thread Time-limited test -> (continuation of Deferred@857602730 after retry RPC@1847373344))) after lookup
18:29:43.942 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:43.943 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:43.943 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:43.943 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:43.982 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:43.982 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:43.982 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [22ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [24ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [25ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [25ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [41ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [43ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [44ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [44ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [82ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1321678960(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17abced8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@857602730 after retry RPC@1847373344) -> (continuation of Deferred@1484442352 after retry RPC@1402737265), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4eb36e7 -> wakeup thread Time-limited test -> (continuation of Deferred@857602730 after retry RPC@1847373344) -> (continuation of Deferred@1484442352 after retry RPC@1402737265))) after lookup
18:29:43.983 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:43.984 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:43.984 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:43.984 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:44.101 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:44.101 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:44.101 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=202), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [22ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [24ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [25ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [25ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [41ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [43ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [44ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [44ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [82ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [84ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [85ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [85ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [199ms] refreshing cache from master, [199ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [201ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1321678960(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17abced8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@857602730 after retry RPC@1847373344) -> (continuation of Deferred@1484442352 after retry RPC@1402737265) -> (continuation of Deferred@903672468 after retry RPC@106983876), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4eb36e7 -> wakeup thread Time-limited test -> (continuation of Deferred@857602730 after retry RPC@1847373344) -> (continuation of Deferred@1484442352 after retry RPC@1402737265) -> (continuation of Deferred@903672468 after retry RPC@106983876))) after lookup
18:29:44.102 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:44.103 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:44.103 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:44.103 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:44.381 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:44.382 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)[L],58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103),d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:44.382 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=482), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [22ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [24ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [25ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [25ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [41ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [43ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [44ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [44ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [82ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [84ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [85ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [85ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [199ms] refreshing cache from master, [199ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [201ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [202ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [204ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [204ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [479ms] refreshing cache from master, [479ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [482ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1321678960(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17abced8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@857602730 after retry RPC@1847373344) -> (continuation of Deferred@1484442352 after retry RPC@1402737265) -> (continuation of Deferred@903672468 after retry RPC@106983876) -> (continuation of Deferred@706376997 after retry RPC@1404034223), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4eb36e7 -> wakeup thread Time-limited test -> (continuation of Deferred@857602730 after retry RPC@1847373344) -> (continuation of Deferred@1484442352 after retry RPC@1402737265) -> (continuation of Deferred@903672468 after retry RPC@106983876) -> (continuation of Deferred@706376997 after retry RPC@1404034223))) after lookup
18:29:44.382 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] connecting to peer
18:29:44.383 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:44.383 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] cleaning up while in state CONNECTING due to: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:44.384 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399) for tablet fac231686ac94760b2e038da9aa2f387: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399
18:29:45.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.272667 23100 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 37ebb4c0266644fa98fa0851dac80a72)
18:29:45.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.272889 23100 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:45.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.273663 23100 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:45.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:45.276393 22781 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:45.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.282183 23033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "58924cf836aa406bbedb44253800ef1e" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" is_pre_election: true
18:29:45.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.282724 23033 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58924cf836aa406bbedb44253800ef1e in term 9.
18:29:45.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.283761 22780 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58924cf836aa406bbedb44253800ef1e, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 37ebb4c0266644fa98fa0851dac80a72
18:29:45.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.284150 23100 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 10 FOLLOWER]: Leader pre-election won for term 11
18:29:45.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.284287 23100 raft_consensus.cc:493] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 37ebb4c0266644fa98fa0851dac80a72)
18:29:45.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.284381 23100 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 10 FOLLOWER]: Advancing to term 11
18:29:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.287495 23100 raft_consensus.cc:515] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.288250 23100 leader_election.cc:290] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 11 election: Requested vote from peers 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399), d202cde0b06b4bff9c06194f4b40ac5d (127.20.55.129:43419)
18:29:45.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.288865 23033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac231686ac94760b2e038da9aa2f387" candidate_uuid: "58924cf836aa406bbedb44253800ef1e" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "d202cde0b06b4bff9c06194f4b40ac5d"
18:29:45.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.289067 23033 raft_consensus.cc:3060] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 9 FOLLOWER]: Advancing to term 11
18:29:45.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:45.289727 22781 leader_election.cc:336] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111)
18:29:45.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.293443 23033 raft_consensus.cc:2468] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58924cf836aa406bbedb44253800ef1e in term 11.
18:29:45.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.293861 22780 leader_election.cc:304] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58924cf836aa406bbedb44253800ef1e, d202cde0b06b4bff9c06194f4b40ac5d; no voters: 37ebb4c0266644fa98fa0851dac80a72
18:29:45.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.294153 23100 raft_consensus.cc:2804] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 11 FOLLOWER]: Leader election won for term 11
18:29:45.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.294600 23100 raft_consensus.cc:697] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [term 11 LEADER]: Becoming Leader. State: Replica: 58924cf836aa406bbedb44253800ef1e, State: Running, Role: LEADER
18:29:45.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.295095 23100 consensus_queue.cc:237] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:45.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.298862 20815 catalog_manager.cc:5696] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e reported cstate change: term changed from 10 to 11, leader changed from 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130) to 58924cf836aa406bbedb44253800ef1e (127.20.55.131). New cstate: current_term: 11 leader_uuid: "58924cf836aa406bbedb44253800ef1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } health_report { overall_health: HEALTHY } } }
18:29:45.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:45.745847 22781 consensus_peers.cc:597] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e -> Peer 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399): Couldn't send request to peer 37ebb4c0266644fa98fa0851dac80a72. Status: Network error: Client connection negotiation failed: client connection to 127.20.55.130:33399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:29:45.802 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet fac231686ac94760b2e038da9aa2f387 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685' with partition [<start>, <end>)
18:29:45.802 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=fac231686ac94760b2e038da9aa2f387@[37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399),58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)[L],d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)]}]
18:29:45.802 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=fac231686ac94760b2e038da9aa2f387, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1903), Traces: [0ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [2ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [2ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [22ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [24ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [25ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [25ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [41ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [43ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [44ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [44ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [82ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [84ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [85ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [85ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [199ms] refreshing cache from master, [199ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [201ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [202ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [204ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [204ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [479ms] refreshing cache from master, [479ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [482ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, [483ms] sending RPC to server 37ebb4c0266644fa98fa0851dac80a72, [484ms] received response from server 37ebb4c0266644fa98fa0851dac80a72: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [485ms] delaying RPC due to: Network error: Failed to connect to peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399): Connection refused: /127.20.55.130:33399, [1900ms] refreshing cache from master, [1900ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.55.189:45691, [1902ms] Sub RPC GetTableLocations: received response from server master-127.20.55.189:45691: OK, deferred=Deferred@1321678960(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007f861436b560@17abced8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@857602730 after retry RPC@1847373344) -> (continuation of Deferred@1484442352 after retry RPC@1402737265) -> (continuation of Deferred@903672468 after retry RPC@106983876) -> (continuation of Deferred@706376997 after retry RPC@1404034223) -> (continuation of Deferred@1919738518 after retry RPC@1602945744), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4eb36e7 -> wakeup thread Time-limited test -> (continuation of Deferred@857602730 after retry RPC@1847373344) -> (continuation of Deferred@1484442352 after retry RPC@1402737265) -> (continuation of Deferred@903672468 after retry RPC@106983876) -> (continuation of Deferred@706376997 after retry RPC@1404034223) -> (continuation of Deferred@1919738518 after retry RPC@1602945744))) after lookup
18:29:45.803 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] connecting to peer
18:29:45.804 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] Successfully connected to peer
18:29:45.819 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0x79f68009, L:/127.0.0.1:57862 - R:/127.20.55.131:42103] using TOKEN/null
18:29:45.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.819952 23033 raft_consensus.cc:1275] T fac231686ac94760b2e038da9aa2f387 P d202cde0b06b4bff9c06194f4b40ac5d [term 11 FOLLOWER]: Refusing update from remote peer 58924cf836aa406bbedb44253800ef1e: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
18:29:45.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.820864 23100 consensus_queue.cc:1048] T fac231686ac94760b2e038da9aa2f387 P 58924cf836aa406bbedb44253800ef1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
18:29:45.863 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.55.130:33399
18:29:45.863 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:45.864203 20702 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/logs
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:29:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.130:33399
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.130
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42109
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.130
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:45.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:29:46.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:46.194777 23128 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
18:29:46.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:46.195115 23128 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:29:46.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:46.195240 23128 flags.cc:432] Enabled unsafe flag: --never_fsync=true
18:29:46.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:46.203433 23128 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
18:29:46.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:46.203655 23128 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.55.130
18:29:46.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.213680 23128 tablet_server_runner.cc:78] Tablet server non-default flags:
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.55.148:39037
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.55.130:33399
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.55.130
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42109
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.55.190:43499,127.20.55.189:45691,127.20.55.188:35241
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.55.130
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/logs
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
18:29:46.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3ddc7d4404ac90c834b1cdd020973c286611e9b0
18:29:46.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:29:46.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 18:01:24 UTC on bdcb31816ec0
18:29:46.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16870
18:29:46.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:29:46.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.215756 23128 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:29:46.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.217610 23128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
18:29:46.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:46.227322 23133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:46.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:46.227293 23134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:46.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:46.228396 23136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:29:46.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.228498 23128 server_base.cc:1061] running on GCE node
18:29:46.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.229137 23128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:29:46.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.230114 23128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:29:46.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.231343 23128 hybrid_clock.cc:648] HybridClock initialized: now 1783535386231282 us; error 47 us; skew 500 ppm
18:29:46.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.231690 23128 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:29:46.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.233922 23128 webserver.cc:534] Webserver started at http://127.20.55.130:42109/ using document root <none> and password file <none>
18:29:46.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.234779 23128 fs_manager.cc:362] Metadata directory not provided
18:29:46.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.234917 23128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:29:46.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.240506 23128 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
18:29:46.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.243528 23142 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:29:46.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.244877 23128 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
18:29:46.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.245025 23128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data,/tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:46.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "37ebb4c0266644fa98fa0851dac80a72"
18:29:46.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 18:29:12 on dist-test-slave-4hqz"
18:29:46.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.245697 23128 fs_report.cc:389] FS layout report
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/wal
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/data
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:29:46.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:29:46.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.261855 23128 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
18:29:46.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.262676 23128 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:29:46.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.262912 23128 kserver.cc:163] Server-wide thread pool size limit: 3276
18:29:46.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.263787 23128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
18:29:46.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.265812 23149 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
18:29:46.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.271638 23128 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:29:46.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.271719 23128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
18:29:46.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.271813 23128 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
18:29:46.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.274700 23128 ts_tablet_manager.cc:616] Registered 1 tablets
18:29:46.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.274784 23128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
18:29:46.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.275264 23149 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap starting.
18:29:46.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.300496 23149 log.cc:826] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Log is configured to *not* fsync() on all Append() calls
18:29:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.351248 23128 rpc_server.cc:307] RPC server started. Bound to: 127.20.55.130:33399
18:29:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.351318 23259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.55.130:33399 every 8 connection(s)
18:29:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.351799 23149 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:29:46.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.352417 23149 tablet_bootstrap.cc:492] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Bootstrap complete.
18:29:46.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.352941 23128 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknrBYU1/test-tmp/mini-kudu-cluster6357081869377392917/ts-1/data/info.pb
18:29:46.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.353797 23149 ts_tablet_manager.cc:1403] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Time spent bootstrapping tablet: real 0.079s	user 0.061s	sys 0.012s
18:29:46.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.362366 20702 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu as pid 23128
18:29:46.363 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:29:46.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.362742 23149 raft_consensus.cc:359] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:46.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.364053 23149 raft_consensus.cc:740] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ebb4c0266644fa98fa0851dac80a72, State: Initialized, Role: FOLLOWER
18:29:46.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.365237 23149 consensus_queue.cc:260] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37ebb4c0266644fa98fa0851dac80a72" member_type: VOTER last_known_addr { host: "127.20.55.130" port: 33399 } } peers { permanent_uuid: "d202cde0b06b4bff9c06194f4b40ac5d" member_type: VOTER last_known_addr { host: "127.20.55.129" port: 43419 } } peers { permanent_uuid: "58924cf836aa406bbedb44253800ef1e" member_type: VOTER last_known_addr { host: "127.20.55.131" port: 42103 } }
18:29:46.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.373708 23149 ts_tablet_manager.cc:1434] T fac231686ac94760b2e038da9aa2f387 P 37ebb4c0266644fa98fa0851dac80a72: Time spent starting tablet: real 0.020s	user 0.010s	sys 0.006s
18:29:46.381 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:46.396 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783535353685, tablet=fac231686ac94760b2e038da9aa2f387, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:29:46.400 [DEBUG - main] (Connection.java:575) [peer master-127.20.55.188:35241(127.20.55.188:35241)] disconnecting while in state READY
18:29:46.401 [DEBUG - main] (Connection.java:575) [peer d202cde0b06b4bff9c06194f4b40ac5d(127.20.55.129:43419)] disconnecting while in state TERMINATED
18:29:46.404 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.20.55.188:35241(127.20.55.188:35241)] handling channelInactive
18:29:46.404 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.20.55.188:35241(127.20.55.188:35241)] cleaning up while in state READY due to: connection closed
18:29:46.404 [DEBUG - main] (Connection.java:575) [peer master-127.20.55.189:45691(127.20.55.189:45691)] disconnecting while in state READY
18:29:46.405 [DEBUG - main] (Connection.java:575) [peer master-127.20.55.190:43499(127.20.55.190:43499)] disconnecting while in state READY
18:29:46.405 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.20.55.189:45691(127.20.55.189:45691)] handling channelInactive
18:29:46.405 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.20.55.189:45691(127.20.55.189:45691)] cleaning up while in state READY due to: connection closed
18:29:46.405 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.20.55.190:43499(127.20.55.190:43499)] handling channelInactive
18:29:46.406 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.20.55.190:43499(127.20.55.190:43499)] cleaning up while in state READY due to: connection closed
18:29:46.407 [DEBUG - main] (Connection.java:575) [peer 37ebb4c0266644fa98fa0851dac80a72(127.20.55.130:33399)] disconnecting while in state TERMINATED
18:29:46.407 [DEBUG - main] (Connection.java:575) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] disconnecting while in state READY
18:29:46.408 [DEBUG - kudu-nio-3] (AsyncKuduClient.java:2704) Releasing all remaining resources
18:29:46.409 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] handling channelInactive
18:29:46.409 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 58924cf836aa406bbedb44253800ef1e(127.20.55.131:42103)] cleaning up while in state READY due to: connection closed
18:29:46.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.413910 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 22948
18:29:46.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.438457 23261 heartbeater.cc:344] Connected to a master server at 127.20.55.190:43499
18:29:46.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.438814 23261 heartbeater.cc:461] Registering TS with master...
18:29:46.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.440030 23261 heartbeater.cc:507] Master 127.20.55.190:43499 requested a full tablet report, sending...
18:29:46.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.443849 23075 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:46.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.447875 23260 heartbeater.cc:344] Connected to a master server at 127.20.55.188:35241
18:29:46.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.448000 23260 heartbeater.cc:461] Registering TS with master...
18:29:46.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.448462 23260 heartbeater.cc:507] Master 127.20.55.188:35241 requested a full tablet report, sending...
18:29:46.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.449566 20743 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:46.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.451115 20887 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.452894 23262 heartbeater.cc:344] Connected to a master server at 127.20.55.189:45691
18:29:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.453071 23262 heartbeater.cc:461] Registering TS with master...
18:29:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.453590 23262 heartbeater.cc:507] Master 127.20.55.189:45691 requested a full tablet report, sending...
18:29:46.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.454815 20815 ts_manager.cc:194] Re-registered known tserver with Master: 37ebb4c0266644fa98fa0851dac80a72 (127.20.55.130:33399)
18:29:46.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.455951 20815 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.55.130:42543
18:29:46.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.458137 23262 heartbeater.cc:499] Master 127.20.55.189:45691 was elected leader, sending a full tablet report...
18:29:46.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.588479 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 22948
18:29:46.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.606685 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 23128
18:29:46.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.616807 23255 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:46.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.708505 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 23128
18:29:46.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.726769 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 22764
18:29:46.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.735426 22890 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:46.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.871819 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 22764
18:29:46.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:46.891990 20730 connection.cc:573] server connection from 127.20.55.131:55807 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:29:46.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 18:29:46.892000 20802 connection.cc:573] server connection from 127.20.55.131:55861 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:29:46.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.892426 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 20713
18:29:46.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:46.893623 20774 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:47.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:47.010636 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 20713
18:29:47.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:47.035521 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 20783
18:29:47.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:47.040807 20846 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:47.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:47.160326 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 20783
18:29:47.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:47.182928 20702 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu pid 20853
18:29:47.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:47.184111 20917 generic_service.cc:196] Checking for leaks (request via RPC)
18:29:47.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 18:29:47.298612 20702 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasknrBYU1/build/asan/bin/kudu with pid 20853
18:29:47.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T18:29:47Z chronyd exiting

Time: 38.589

OK (1 test)

