JUnit version 4.13.2
.14:09:38.268 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1783519778261
14:09:38.322 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
14:09:38.380 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
14:09:38.392 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskGaLwZW/test-tmp
14:09:38.397 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/release/bin
14:09:38.400 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskGaLwZW/java/build/dist-test/../../../build/release/bin/kudu, test, mini_cluster, --serialization=pb]
14:09:38.597 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

14:09:38.614 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:09:38.616 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

14:09:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.617254 29698 mini_oidc.cc:150] Starting JWKS server
14:09:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.617972 29698 webserver.cc:533] Webserver started at http://127.0.0.1:39953/ using document root <none> and password file <none>
14:09:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.618072 29698 mini_oidc.cc:186] Starting OIDC Discovery server
14:09:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.618222 29698 webserver.cc:533] Webserver started at http://127.0.0.1:38519/ using document root <none> and password file <none>
14:09:38.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T14:09:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
14:09:38.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T14:09:38Z Disabled control of system clock
14:09:38.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.630702 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:38.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:38.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/wal
14:09:38.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/data
14:09:38.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:38.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:38.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:38.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:38.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:38.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:38.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/logs
14:09:38.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/data/info.pb
14:09:38.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:38.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:38.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:38.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:38.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:38.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:38.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:38.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:38.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:09:38.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:38.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:09:38.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:09:38.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.190:45183
14:09:38.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.190
14:09:38.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:38.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:38.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:38.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:38.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:38.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:09:38.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:38.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:09:38.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39953/jwks.json with env {}
14:09:38.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.713565 29710 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:38.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.713735 29710 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:38.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.713755 29710 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:38.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.715642 29710 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:09:38.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.715724 29710 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:38.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.715749 29710 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:09:38.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.715765 29710 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:09:38.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.715785 29710 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:09:38.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.715801 29710 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:39953/jwks.json
14:09:38.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.717694 29710 master_runner.cc:387] Master server non-default flags:
14:09:38.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:38.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:38.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:38.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:38.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/data
14:09:38.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/wal
14:09:38.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:09:38.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:38.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:38.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:38.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:09:38.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.190:45183
14:09:38.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:09:38.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:38.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:09:38.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39953/jwks.json
14:09:38.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:38.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:38.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/data/info.pb
14:09:38.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.190
14:09:38.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:38.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:38.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.29710
14:09:38.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:38.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:38.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:38.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:38.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/logs
14:09:38.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:38.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:38.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:38.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:09:38.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:38.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:38.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:38.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:38.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:38.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.717875 29710 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:38.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.718084 29710 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:38.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.720624 29715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:38.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.720630 29716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:38.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.720737 29718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:38.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.721519 29710 server_base.cc:1061] running on GCE node
14:09:38.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.721706 29710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:38.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.721922 29710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:38.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.723042 29710 hybrid_clock.cc:648] HybridClock initialized: now 1783519778723024 us; error 35 us; skew 500 ppm
14:09:38.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.724319 29710 webserver.cc:533] Webserver started at http://127.29.0.190:35051/ using document root <none> and password file <none>
14:09:38.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.724469 29710 fs_manager.cc:362] Metadata directory not provided
14:09:38.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.724502 29710 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:38.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.724573 29710 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:09:38.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.725566 29710 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/data/instance:
14:09:38.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bb8aca011aea44e2a93530bdd3dcdcf5"
14:09:38.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:38 on dist-test-slave-0nsb"
14:09:38.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.725926 29710 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/wal/instance:
14:09:38.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bb8aca011aea44e2a93530bdd3dcdcf5"
14:09:38.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:38 on dist-test-slave-0nsb"
14:09:38.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.727399 29710 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.001s
14:09:38.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.728317 29724 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:38.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.728482 29710 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
14:09:38.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.728547 29710 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/wal
14:09:38.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bb8aca011aea44e2a93530bdd3dcdcf5"
14:09:38.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:38 on dist-test-slave-0nsb"
14:09:38.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.728600 29710 fs_report.cc:389] FS layout report
14:09:38.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:38.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/wal
14:09:38.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/wal
14:09:38.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/data/data
14:09:38.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:38.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:38.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:38.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:38.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:38.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:38.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:38.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:38.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:38.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:38.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:09:38.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.752831 29710 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:09:38.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.753096 29710 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:09:38.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.753177 29710 kserver.cc:163] Server-wide thread pool size limit: 3276
14:09:38.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.757511 29710 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.190:45183
14:09:38.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.757550 29776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.190:45183 every 8 connection(s)
14:09:38.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.757927 29710 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/data/info.pb
14:09:38.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.758599 29777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:09:38.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.760406 29777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } has no permanent_uuid. Determining permanent_uuid...
14:09:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.766848 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 29710
14:09:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.766932 29698 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-0/wal/instance
14:09:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.767604 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/wal
14:09:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/data
14:09:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/logs
14:09:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/data/info.pb
14:09:38.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:38.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:38.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:38.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:38.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:38.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:38.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:09:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:09:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:09:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.189:46475
14:09:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.189
14:09:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:38.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:38.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:09:38.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:38.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:09:38.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39953/jwks.json with env {}
14:09:38.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.769709 29777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } has no permanent_uuid. Determining permanent_uuid...
14:09:38.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.770025 29725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.189:46475: connect: Connection refused (error 111)
14:09:38.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.770484 29777 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.0.189:46475: Network error: Client connection negotiation failed: client connection to 127.29.0.189:46475: connect: Connection refused (error 111)
14:09:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.819716 29777 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } attempt: 1
14:09:38.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.820596 29777 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.0.189:46475: Network error: Client connection negotiation failed: client connection to 127.29.0.189:46475: connect: Connection refused (error 111)
14:09:38.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.849170 29781 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:38.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.849345 29781 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:38.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.849366 29781 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:38.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.851508 29781 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:09:38.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.851564 29781 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:38.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.851600 29781 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:09:38.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.851615 29781 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:09:38.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.851629 29781 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:09:38.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.851642 29781 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:39953/jwks.json
14:09:38.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.854135 29781 master_runner.cc:387] Master server non-default flags:
14:09:38.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:38.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:38.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:38.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:38.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/data
14:09:38.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/wal
14:09:38.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:09:38.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:38.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:38.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:38.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:09:38.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.189:46475
14:09:38.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:09:38.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:38.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:09:38.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39953/jwks.json
14:09:38.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:38.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:38.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/data/info.pb
14:09:38.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.189
14:09:38.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:38.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:38.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.29781
14:09:38.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:38.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:38.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:38.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:38.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/logs
14:09:38.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:38.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:38.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:38.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:09:38.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:38.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:38.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:38.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:38.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:38.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.854382 29781 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:38.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.854627 29781 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:38.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.857309 29788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:38.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.857923 29789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:38.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.858031 29791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:38.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.858264 29781 server_base.cc:1061] running on GCE node
14:09:38.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.858491 29781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:38.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.858724 29781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:38.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.859939 29781 hybrid_clock.cc:648] HybridClock initialized: now 1783519778859906 us; error 46 us; skew 500 ppm
14:09:38.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.861922 29781 webserver.cc:533] Webserver started at http://127.29.0.189:40035/ using document root <none> and password file <none>
14:09:38.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.862123 29781 fs_manager.cc:362] Metadata directory not provided
14:09:38.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.862174 29781 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:38.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.862277 29781 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:09:38.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.863404 29781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/data/instance:
14:09:38.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dbb2ceb0da474aa0bb1f8de2b183013a"
14:09:38.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:38 on dist-test-slave-0nsb"
14:09:38.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.863813 29781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/wal/instance:
14:09:38.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dbb2ceb0da474aa0bb1f8de2b183013a"
14:09:38.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:38 on dist-test-slave-0nsb"
14:09:38.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.865465 29781 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
14:09:38.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.866516 29797 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:38.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.866701 29781 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
14:09:38.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.866752 29781 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/wal
14:09:38.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dbb2ceb0da474aa0bb1f8de2b183013a"
14:09:38.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:38 on dist-test-slave-0nsb"
14:09:38.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.866803 29781 fs_report.cc:389] FS layout report
14:09:38.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:38.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/wal
14:09:38.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/wal
14:09:38.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/data/data
14:09:38.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:38.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:38.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:38.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:38.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:38.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:38.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:38.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:38.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:38.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:38.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:09:38.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.888764 29777 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } attempt: 2
14:09:38.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.889489 29777 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.0.189:46475: Network error: Client connection negotiation failed: client connection to 127.29.0.189:46475: connect: Connection refused (error 111)
14:09:38.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.897579 29781 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:09:38.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.897874 29781 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:09:38.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.897982 29781 kserver.cc:163] Server-wide thread pool size limit: 3276
14:09:38.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.902495 29781 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.189:46475
14:09:38.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.902552 29849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.189:46475 every 8 connection(s)
14:09:38.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.902906 29781 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/data/info.pb
14:09:38.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.903496 29850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:09:38.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.905256 29850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } has no permanent_uuid. Determining permanent_uuid...
14:09:38.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.912273 29850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } has no permanent_uuid. Determining permanent_uuid...
14:09:38.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.912622 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 29781
14:09:38.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.912694 29698 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-1/wal/instance
14:09:38.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.913154 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:38.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:38.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/wal
14:09:38.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/data
14:09:38.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:38.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:38.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:38.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:38.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:38.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:38.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/logs
14:09:38.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/data/info.pb
14:09:38.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:38.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:38.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:38.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:38.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:38.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:38.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:38.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:38.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:09:38.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:38.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:09:38.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:09:38.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.188:40013
14:09:38.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.188
14:09:38.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:38.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:38.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:38.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:38.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:38.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:09:38.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:38.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:09:38.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39953/jwks.json with env {}
14:09:38.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.916433 29850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } has no permanent_uuid. Determining permanent_uuid...
14:09:38.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.916886 29799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.188:40013: connect: Connection refused (error 111)
14:09:38.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.917277 29850 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.0.188:40013: Network error: Client connection negotiation failed: client connection to 127.29.0.188:40013: connect: Connection refused (error 111)
14:09:38.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.966471 29850 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } attempt: 1
14:09:38.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.967501 29850 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.0.188:40013: Network error: Client connection negotiation failed: client connection to 127.29.0.188:40013: connect: Connection refused (error 111)
14:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.980651 29777 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } attempt: 3
14:09:38.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.986235 29777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } has no permanent_uuid. Determining permanent_uuid...
14:09:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.987004 29777 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.0.188:40013: Network error: Client connection negotiation failed: client connection to 127.29.0.188:40013: connect: Connection refused (error 111)
14:09:38.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.996241 29854 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:38.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.996415 29854 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:38.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.996434 29854 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:38.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.997923 29854 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:09:38.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.997974 29854 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:38.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.997987 29854 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:09:38.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.997999 29854 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:09:38.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.998011 29854 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:09:38.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:38.998023 29854 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:39953/jwks.json
14:09:38.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.999581 29854 master_runner.cc:387] Master server non-default flags:
14:09:38.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/data
14:09:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/wal
14:09:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:09:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:09:39.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.188:40013
14:09:39.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:09:39.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:39.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:09:39.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39953/jwks.json
14:09:39.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:39.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/data/info.pb
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.188
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.29854
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/logs
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:09:39.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:39.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:39.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:39.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:39.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:39.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:38.999823 29854 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:39.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.000013 29854 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:39.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.005160 29861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.005191 29864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.005259 29862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.006951 29854 server_base.cc:1061] running on GCE node
14:09:39.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.007249 29854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:39.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.007573 29854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:39.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.008754 29854 hybrid_clock.cc:648] HybridClock initialized: now 1783519779008733 us; error 38 us; skew 500 ppm
14:09:39.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.010020 29854 webserver.cc:533] Webserver started at http://127.29.0.188:37977/ using document root <none> and password file <none>
14:09:39.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.010252 29854 fs_manager.cc:362] Metadata directory not provided
14:09:39.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.010311 29854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:39.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.010421 29854 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:09:39.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.011420 29854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/data/instance:
14:09:39.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b2f821711f1744f1a9125c08045909e2"
14:09:39.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.015192 29854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/wal/instance:
14:09:39.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b2f821711f1744f1a9125c08045909e2"
14:09:39.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.016884 29854 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
14:09:39.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.017803 29870 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:39.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.018004 29854 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
14:09:39.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.018075 29854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/wal
14:09:39.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b2f821711f1744f1a9125c08045909e2"
14:09:39.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.018137 29854 fs_report.cc:389] FS layout report
14:09:39.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:39.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/wal
14:09:39.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/wal
14:09:39.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/data/data
14:09:39.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:39.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:39.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:39.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:39.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:39.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:39.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:39.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:39.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:39.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:39.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:09:39.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.018148 29777 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } attempt: 1
14:09:39.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.018895 29777 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.0.188:40013: Network error: Client connection negotiation failed: client connection to 127.29.0.188:40013: connect: Connection refused (error 111)
14:09:39.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.029193 29854 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:09:39.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.029506 29854 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:09:39.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.029623 29854 kserver.cc:163] Server-wide thread pool size limit: 3276
14:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.034703 29854 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.188:40013
14:09:39.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.034741 29922 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.188:40013 every 8 connection(s)
14:09:39.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.035136 29854 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/data/info.pb
14:09:39.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.035703 29850 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } attempt: 2
14:09:39.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.035813 29923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:09:39.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.038141 29923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } has no permanent_uuid. Determining permanent_uuid...
14:09:39.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.038714 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 29854
14:09:39.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.038789 29698 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/master-2/wal/instance
14:09:39.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.041540 29850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a: Bootstrap starting.
14:09:39.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.042287 29850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a: Neither blocks nor log segments found. Creating new log.
14:09:39.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.042539 29850 log.cc:826] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a: Log is configured to *not* fsync() on all Append() calls
14:09:39.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.043184 29850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a: No bootstrap required, opened a new log
14:09:39.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.043834 29923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } has no permanent_uuid. Determining permanent_uuid...
14:09:39.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.044871 29850 raft_consensus.cc:359] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } }
14:09:39.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.044996 29850 raft_consensus.cc:385] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:09:39.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.045032 29850 raft_consensus.cc:740] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2ceb0da474aa0bb1f8de2b183013a, State: Initialized, Role: FOLLOWER
14:09:39.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.045123 29850 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } }
14:09:39.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.045827 29850 sys_catalog.cc:565] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [sys.catalog]: configured and running, proceeding with master startup.
14:09:39.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.046665 29931 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } } }
14:09:39.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.047015 29931 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [sys.catalog]: This master's current role is: FOLLOWER
14:09:39.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.048367 29942 catalog_manager.cc:1594] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:09:39.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.049010 29942 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:09:39.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.049520 29923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } has no permanent_uuid. Determining permanent_uuid...
14:09:39.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.048095 29781 master_runner.cc:187] Error getting master registration for 127.29.0.190:45183: OK, instance_id { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" instance_seqno: 1783519778756396 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:09:39.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.049875 29781 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:09:39.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.055559 29923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2: Bootstrap starting.
14:09:39.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.056345 29923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2: Neither blocks nor log segments found. Creating new log.
14:09:39.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.056542 29923 log.cc:826] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2: Log is configured to *not* fsync() on all Append() calls
14:09:39.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.060325 29923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2: No bootstrap required, opened a new log
14:09:39.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.061897 29923 raft_consensus.cc:359] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } }
14:09:39.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.062006 29923 raft_consensus.cc:385] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:09:39.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.062029 29923 raft_consensus.cc:740] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f821711f1744f1a9125c08045909e2, State: Initialized, Role: FOLLOWER
14:09:39.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.062129 29923 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } }
14:09:39.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.062399 29946 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } } }
14:09:39.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.062462 29923 sys_catalog.cc:565] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [sys.catalog]: configured and running, proceeding with master startup.
14:09:39.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.062475 29946 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [sys.catalog]: This master's current role is: FOLLOWER
14:09:39.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.064596 29957 catalog_manager.cc:1594] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:09:39.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.064656 29957 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:09:39.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.064869 29854 master_runner.cc:187] Error getting master registration for 127.29.0.190:45183: OK, instance_id { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" instance_seqno: 1783519778756396 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:09:39.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.066279 29854 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:09:39.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.094061 29777 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } attempt: 2
14:09:39.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.099113 29777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5: Bootstrap starting.
14:09:39.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.099908 29777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5: Neither blocks nor log segments found. Creating new log.
14:09:39.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.100469 29777 log.cc:826] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5: Log is configured to *not* fsync() on all Append() calls
14:09:39.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.102453 29777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5: No bootstrap required, opened a new log
14:09:39.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.105479 29777 raft_consensus.cc:359] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } }
14:09:39.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.105633 29777 raft_consensus.cc:385] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:09:39.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.105665 29777 raft_consensus.cc:740] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8aca011aea44e2a93530bdd3dcdcf5, State: Initialized, Role: FOLLOWER
14:09:39.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.105819 29777 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } }
14:09:39.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.107084 29777 sys_catalog.cc:565] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [sys.catalog]: configured and running, proceeding with master startup.
14:09:39.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.111655 29710 master_runner.cc:264] No leader master found from master bb8aca011aea44e2a93530bdd3dcdcf5
14:09:39.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.111929 29710 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:09:39.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.115806 29960 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } } }
14:09:39.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.115903 29960 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [sys.catalog]: This master's current role is: FOLLOWER
14:09:39.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.118173 29971 catalog_manager.cc:1594] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:09:39.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.118530 29971 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:09:39.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.139422 29931 raft_consensus.cc:493] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:09:39.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.139580 29931 raft_consensus.cc:515] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } }
14:09:39.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.139911 29931 leader_election.cc:290] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb8aca011aea44e2a93530bdd3dcdcf5 (127.29.0.190:45183), b2f821711f1744f1a9125c08045909e2 (127.29.0.188:40013)
14:09:39.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.140177 29897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f821711f1744f1a9125c08045909e2" is_pre_election: true
14:09:39.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.140240 29751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" is_pre_election: true
14:09:39.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.140345 29897 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb2ceb0da474aa0bb1f8de2b183013a in term 0.
14:09:39.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.140375 29751 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb2ceb0da474aa0bb1f8de2b183013a in term 0.
14:09:39.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.140543 29799 leader_election.cc:304] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2f821711f1744f1a9125c08045909e2, dbb2ceb0da474aa0bb1f8de2b183013a; no voters: 
14:09:39.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.140687 29931 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [term 0 FOLLOWER]: Leader pre-election won for term 1
14:09:39.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.140731 29931 raft_consensus.cc:493] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:09:39.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.140754 29931 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [term 0 FOLLOWER]: Advancing to term 1
14:09:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.148710 29803 tablet.cc:2406] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:09:39.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.215873 29931 raft_consensus.cc:515] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } }
14:09:39.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.216102 29931 leader_election.cc:290] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [CANDIDATE]: Term 1 election: Requested vote from peers bb8aca011aea44e2a93530bdd3dcdcf5 (127.29.0.190:45183), b2f821711f1744f1a9125c08045909e2 (127.29.0.188:40013)
14:09:39.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.218053 29751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5"
14:09:39.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.218158 29751 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [term 0 FOLLOWER]: Advancing to term 1
14:09:39.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.219146 29897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f821711f1744f1a9125c08045909e2"
14:09:39.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.219223 29897 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [term 0 FOLLOWER]: Advancing to term 1
14:09:39.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.253901 29730 tablet.cc:2406] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:09:39.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.280244 29876 tablet.cc:2406] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:09:39.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.313479 29751 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb2ceb0da474aa0bb1f8de2b183013a in term 1.
14:09:39.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.313900 29897 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb2ceb0da474aa0bb1f8de2b183013a in term 1.
14:09:39.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.314698 29798 leader_election.cc:304] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb8aca011aea44e2a93530bdd3dcdcf5, dbb2ceb0da474aa0bb1f8de2b183013a; no voters: 
14:09:39.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.314935 29931 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [term 1 FOLLOWER]: Leader election won for term 1
14:09:39.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.315099 29931 raft_consensus.cc:697] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [term 1 LEADER]: Becoming Leader. State: Replica: dbb2ceb0da474aa0bb1f8de2b183013a, State: Running, Role: LEADER
14:09:39.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.315197 29931 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } }
14:09:39.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.315610 29931 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb2ceb0da474aa0bb1f8de2b183013a. Latest consensus state: current_term: 1 leader_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } } }
14:09:39.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.315701 29931 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [sys.catalog]: This master's current role is: LEADER
14:09:39.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.316336 29975 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
14:09:39.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.316555 29975 catalog_manager.cc:1520] Initializing Kudu cluster ID...
14:09:39.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.318087 29897 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [term 1 FOLLOWER]: Refusing update from remote peer dbb2ceb0da474aa0bb1f8de2b183013a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:09:39.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.318153 29751 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [term 1 FOLLOWER]: Refusing update from remote peer dbb2ceb0da474aa0bb1f8de2b183013a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:09:39.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.318357 29973 consensus_queue.cc:1048] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:09:39.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.318434 29931 consensus_queue.cc:1048] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:09:39.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.319375 29960 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb2ceb0da474aa0bb1f8de2b183013a. Latest consensus state: current_term: 1 leader_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } } }
14:09:39.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.319453 29960 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [sys.catalog]: This master's current role is: FOLLOWER
14:09:39.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.319408 29946 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb2ceb0da474aa0bb1f8de2b183013a. Latest consensus state: current_term: 1 leader_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } } }
14:09:39.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.319541 29946 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [sys.catalog]: This master's current role is: FOLLOWER
14:09:39.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.320745 29931 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } } }
14:09:39.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.320817 29931 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [sys.catalog]: This master's current role is: LEADER
14:09:39.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.321213 29960 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } } }
14:09:39.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.321297 29960 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5 [sys.catalog]: This master's current role is: FOLLOWER
14:09:39.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.321666 29976 mvcc.cc:204] Tried to move back new op lower bound from 7305297016084144128 to 7305297016075595776. Current Snapshot: MvccSnapshot[applied={T|T < 7305297016084144128}]
14:09:39.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.321880 29977 mvcc.cc:204] Tried to move back new op lower bound from 7305297016084144128 to 7305297016075595776. Current Snapshot: MvccSnapshot[applied={T|T < 7305297016084144128}]
14:09:39.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.322410 29931 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } } }
14:09:39.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.322474 29975 catalog_manager.cc:1383] Generated new cluster ID: 2549dbcbb33543568daee1f1e7e92f28
14:09:39.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.322516 29975 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
14:09:39.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.322482 29931 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a [sys.catalog]: This master's current role is: LEADER
14:09:39.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.323114 29946 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb8aca011aea44e2a93530bdd3dcdcf5" member_type: VOTER last_known_addr { host: "127.29.0.190" port: 45183 } } peers { permanent_uuid: "dbb2ceb0da474aa0bb1f8de2b183013a" member_type: VOTER last_known_addr { host: "127.29.0.189" port: 46475 } } peers { permanent_uuid: "b2f821711f1744f1a9125c08045909e2" member_type: VOTER last_known_addr { host: "127.29.0.188" port: 40013 } } }
14:09:39.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.323192 29946 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2 [sys.catalog]: This master's current role is: FOLLOWER
14:09:39.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.323263 29978 mvcc.cc:204] Tried to move back new op lower bound from 7305297016084144128 to 7305297016075595776. Current Snapshot: MvccSnapshot[applied={T|T < 7305297016084144128}]
14:09:39.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.337504 29975 catalog_manager.cc:1406] Generated new certificate authority record
14:09:39.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.338188 29975 catalog_manager.cc:1540] Loading token signing keys...
14:09:39.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.345803 29975 catalog_manager.cc:6070] T 00000000000000000000000000000000 P dbb2ceb0da474aa0bb1f8de2b183013a: Generated new TSK 0
14:09:39.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.346004 29975 catalog_manager.cc:1550] Initializing in-progress tserver states...
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.348737 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/logs
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:39.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.129:0
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.129
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.129
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:39.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:39.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:09:39.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.432471 29985 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:39.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.432651 29985 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:39.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.432672 29985 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:39.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.434370 29985 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:39.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.434433 29985 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.129
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.436086 29985 tablet_server_runner.cc:78] Tablet server non-default flags:
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.129:0
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:09:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.129
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.29985
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.129
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/logs
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:39.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:39.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:39.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:09:39.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:39.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:39.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:39.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:39.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:39.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.436276 29985 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:39.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.436455 29985 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:39.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.438900 29993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.438894 29990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.439076 29991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.439848 29985 server_base.cc:1061] running on GCE node
14:09:39.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.440034 29985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:39.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.440276 29985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:39.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.441690 29985 hybrid_clock.cc:648] HybridClock initialized: now 1783519779441668 us; error 37 us; skew 500 ppm
14:09:39.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.443094 29985 webserver.cc:533] Webserver started at http://127.29.0.129:44997/ using document root <none> and password file <none>
14:09:39.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.443301 29985 fs_manager.cc:362] Metadata directory not provided
14:09:39.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.443349 29985 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:39.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.443452 29985 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:09:39.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.444566 29985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/instance:
14:09:39.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2d90e981114c450186889150658715c8"
14:09:39.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.444924 29985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal/instance:
14:09:39.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2d90e981114c450186889150658715c8"
14:09:39.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.446705 29985 fs_manager.cc:696] Time spent creating directory manager: real 0.002s	user 0.000s	sys 0.003s
14:09:39.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.447561 29999 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:39.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.448020 29985 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
14:09:39.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.448087 29985 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:39.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2d90e981114c450186889150658715c8"
14:09:39.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.448146 29985 fs_report.cc:389] FS layout report
14:09:39.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:39.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:39.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:39.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/data
14:09:39.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:39.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:39.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:39.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:39.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:39.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:39.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:39.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:39.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:39.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:39.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:09:39.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.465603 29985 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:09:39.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.465950 29985 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:09:39.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.466079 29985 kserver.cc:163] Server-wide thread pool size limit: 3276
14:09:39.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.466287 29985 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:09:39.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.466714 29985 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:09:39.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.466759 29985 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:09:39.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.466794 29985 ts_tablet_manager.cc:616] Registered 0 tablets
14:09:39.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.466813 29985 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:09:39.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.474027 29985 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.129:37851
14:09:39.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.474113 30112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.129:37851 every 8 connection(s)
14:09:39.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.474458 29985 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:09:39.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.479866 30113 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:09:39.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.479974 30113 heartbeater.cc:461] Registering TS with master...
14:09:39.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.480160 30113 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:09:39.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.480643 29887 ts_manager.cc:194] Registered new tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:09:39.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.480700 30114 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:09:39.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.480739 30114 heartbeater.cc:461] Registering TS with master...
14:09:39.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.480821 30114 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:09:39.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.481143 29741 ts_manager.cc:194] Registered new tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:09:39.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.481391 30115 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:09:39.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.481448 30115 heartbeater.cc:461] Registering TS with master...
14:09:39.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.481508 30115 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:09:39.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.481814 29814 ts_manager.cc:194] Registered new tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:09:39.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.482409 29814 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.129:42353
14:09:39.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.483861 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 29985
14:09:39.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.483924 29698 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal/instance
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.485076 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/logs
14:09:39.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.130:0
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.130
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.130
14:09:39.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:39.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:39.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:39.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:39.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:39.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:09:39.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.564699 30120 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:39.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.564883 30120 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:39.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.564903 30120 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:39.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.566401 30120 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:39.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.566455 30120 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.130
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.568020 30120 tablet_server_runner.cc:78] Tablet server non-default flags:
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.130:0
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.130
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:39.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.30120
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.130
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/logs
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:39.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:39.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.568200 30120 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:39.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.568384 30120 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:39.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.570950 30128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.571002 30125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.571050 30126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.571303 30120 server_base.cc:1061] running on GCE node
14:09:39.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.571489 30120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:39.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.571851 30120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:39.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.572995 30120 hybrid_clock.cc:648] HybridClock initialized: now 1783519779572970 us; error 35 us; skew 500 ppm
14:09:39.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.574174 30120 webserver.cc:533] Webserver started at http://127.29.0.130:33025/ using document root <none> and password file <none>
14:09:39.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.574399 30120 fs_manager.cc:362] Metadata directory not provided
14:09:39.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.574524 30120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:39.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.574637 30120 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:09:39.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.575570 30120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/instance:
14:09:39.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0f958fcb6937451184fba50df6455e4d"
14:09:39.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.575981 30120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal/instance:
14:09:39.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0f958fcb6937451184fba50df6455e4d"
14:09:39.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.577381 30120 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
14:09:39.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.578204 30134 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:39.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.578399 30120 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
14:09:39.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.578482 30120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:39.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0f958fcb6937451184fba50df6455e4d"
14:09:39.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.578540 30120 fs_report.cc:389] FS layout report
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/data
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:39.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:39.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:39.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:39.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:09:39.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.596248 30120 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:09:39.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.596628 30120 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:09:39.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.596755 30120 kserver.cc:163] Server-wide thread pool size limit: 3276
14:09:39.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.597021 30120 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:09:39.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.597433 30120 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:09:39.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.597488 30120 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:09:39.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.597527 30120 ts_tablet_manager.cc:616] Registered 0 tablets
14:09:39.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.597548 30120 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:09:39.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.605082 30120 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.130:34547
14:09:39.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.605141 30247 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.130:34547 every 8 connection(s)
14:09:39.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.605536 30120 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:09:39.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.609745 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 30120
14:09:39.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.609817 29698 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal/instance
14:09:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.611248 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data
14:09:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/logs
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.131:0
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.131
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.131
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:39.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:39.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:09:39.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.614337 30249 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:09:39.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.614468 30249 heartbeater.cc:461] Registering TS with master...
14:09:39.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.614689 30249 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:09:39.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.614476 30250 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:09:39.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.614971 30250 heartbeater.cc:461] Registering TS with master...
14:09:39.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.615059 30250 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:09:39.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.615097 29741 ts_manager.cc:194] Registered new tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:39.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.615291 29814 ts_manager.cc:194] Registered new tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:39.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.615654 29814 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.130:35981
14:09:39.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.615747 30248 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:09:39.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.615799 30248 heartbeater.cc:461] Registering TS with master...
14:09:39.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.616042 30248 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:09:39.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.616322 29887 ts_manager.cc:194] Registered new tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:39.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.693053 30255 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:39.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.693222 30255 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:39.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.693243 30255 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:39.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.694743 30255 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:39.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.694798 30255 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.131
14:09:39.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.696501 30255 tablet_server_runner.cc:78] Tablet server non-default flags:
14:09:39.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:39.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:39.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:39.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:39.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data
14:09:39.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:39.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:39.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:39.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.131:0
14:09:39.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:39.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.131
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.30255
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.131
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/logs
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:09:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.696687 30255 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.696863 30255 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:39.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.699791 30263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.700093 30255 server_base.cc:1061] running on GCE node
14:09:39.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.699869 30261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:39.699812 30260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:39.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.700390 30255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:39.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.700570 30255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:39.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.701685 30255 hybrid_clock.cc:648] HybridClock initialized: now 1783519779701656 us; error 49 us; skew 500 ppm
14:09:39.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.702708 30255 webserver.cc:533] Webserver started at http://127.29.0.131:40737/ using document root <none> and password file <none>
14:09:39.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.702915 30255 fs_manager.cc:362] Metadata directory not provided
14:09:39.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.702960 30255 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:39.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.703030 30255 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:09:39.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.703898 30255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/instance:
14:09:39.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1351807c09ec4b70b1adf098a1d80986"
14:09:39.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.704176 30255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal/instance:
14:09:39.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1351807c09ec4b70b1adf098a1d80986"
14:09:39.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.705677 30255 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
14:09:39.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.706497 30269 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:39.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.707832 30255 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
14:09:39.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.707922 30255 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:39.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1351807c09ec4b70b1adf098a1d80986"
14:09:39.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:39.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.707984 30255 fs_report.cc:389] FS layout report
14:09:39.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:39.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:39.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:39.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/data
14:09:39.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:39.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:39.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:39.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:39.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:39.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:39.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:39.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:39.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:39.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:39.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:09:39.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.731926 30255 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:09:39.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.732204 30255 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:09:39.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.732291 30255 kserver.cc:163] Server-wide thread pool size limit: 3276
14:09:39.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.732472 30255 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:09:39.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.732726 30255 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:09:39.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.732748 30255 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:09:39.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.732769 30255 ts_tablet_manager.cc:616] Registered 0 tablets
14:09:39.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.732780 30255 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:09:39.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.738919 30255 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.131:38303
14:09:39.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.739007 30382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.131:38303 every 8 connection(s)
14:09:39.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.739326 30255 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:09:39.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.745038 30384 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:09:39.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.745095 30383 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:09:39.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.745154 30383 heartbeater.cc:461] Registering TS with master...
14:09:39.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.745157 30384 heartbeater.cc:461] Registering TS with master...
14:09:39.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.745370 30384 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:09:39.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.745370 30383 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:09:39.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.745816 30385 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:09:39.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.745903 30385 heartbeater.cc:461] Registering TS with master...
14:09:39.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.745914 29887 ts_manager.cc:194] Registered new tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:39.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.745998 30385 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:09:39.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.746075 29741 ts_manager.cc:194] Registered new tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:39.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.746199 29814 ts_manager.cc:194] Registered new tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:39.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.746552 29814 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.131:44403
14:09:39.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.747409 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 30255
14:09:39.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.747479 29698 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal/instance
14:09:39.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:39.749858 29698 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
14:09:39.750 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:09:39.752 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

14:09:39.762 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.29.0.190"
      port: 45183
    }
    bound_http_address {
      host: "127.29.0.190"
      port: 35051
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.29.0.189"
      port: 46475
    }
    bound_http_address {
      host: "127.29.0.189"
      port: 40035
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.29.0.188"
      port: 40013
    }
    bound_http_address {
      host: "127.29.0.188"
      port: 37977
    }
  }
}

14:09:39.765 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

14:09:39.768 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.29.0.129"
      port: 37851
    }
    bound_http_address {
      host: "127.29.0.129"
      port: 44997
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.29.0.130"
      port: 34547
    }
    bound_http_address {
      host: "127.29.0.130"
      port: 33025
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.29.0.131"
      port: 38303
    }
    bound_http_address {
      host: "127.29.0.131"
      port: 40737
    }
  }
}

14:09:39.768 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
14:09:39.951 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
14:09:40.008 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
14:09:40.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.065106 29957 catalog_manager.cc:1295] Loaded cluster ID: 2549dbcbb33543568daee1f1e7e92f28
14:09:40.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.065186 29957 catalog_manager.cc:1588] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2: loading cluster ID for follower catalog manager: success
14:09:40.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.066531 29957 catalog_manager.cc:1610] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2: acquiring CA information for follower catalog manager: success
14:09:40.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.066838 29957 catalog_manager.cc:1638] T 00000000000000000000000000000000 P b2f821711f1744f1a9125c08045909e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:09:40.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.119283 29971 catalog_manager.cc:1295] Loaded cluster ID: 2549dbcbb33543568daee1f1e7e92f28
14:09:40.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.119360 29971 catalog_manager.cc:1588] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5: loading cluster ID for follower catalog manager: success
14:09:40.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.120667 29971 catalog_manager.cc:1610] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5: acquiring CA information for follower catalog manager: success
14:09:40.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.121013 29971 catalog_manager.cc:1638] T 00000000000000000000000000000000 P bb8aca011aea44e2a93530bdd3dcdcf5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:09:40.165 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.29.0.188:40013(127.29.0.188:40013)] connecting to peer
14:09:40.221 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.29.0.189:46475(127.29.0.189:46475)] connecting to peer
14:09:40.228 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.29.0.190:45183(127.29.0.190:45183)] connecting to peer
14:09:40.245 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.29.0.189:46475(127.29.0.189:46475)] Successfully connected to peer
14:09:40.250 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.29.0.188:40013(127.29.0.188:40013)] Successfully connected to peer
14:09:40.251 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.29.0.190:45183(127.29.0.190:45183)] Successfully connected to peer
14:09:40.353 [DEBUG - kudu-nio-1] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.29.0.189
14:09:40.353 [DEBUG - kudu-nio-2] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.29.0.190
14:09:40.355 [DEBUG - kudu-nio-0] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.29.0.188
14:09:40.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.483022 30115 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:09:40.590 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0xcfb39c9e, L:/127.0.0.1:43850 - R:/127.29.0.190:45183] using SASL/PLAIN
14:09:40.590 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0xa8896f80, L:/127.0.0.1:50202 - R:/127.29.0.188:40013] using SASL/PLAIN
14:09:40.590 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x0185966f, L:/127.0.0.1:34048 - R:/127.29.0.189:46475] using SASL/PLAIN
14:09:40.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.616472 30250 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:09:40.629 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
14:09:40.630 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=511), Traces: [0ms] refreshing cache from master, [30ms] Sub RPC ConnectToMaster: sending RPC to server master-127.29.0.188:40013, [98ms] Sub RPC ConnectToMaster: sending RPC to server master-127.29.0.189:46475, [103ms] Sub RPC ConnectToMaster: sending RPC to server master-127.29.0.190:45183, [479ms] Sub RPC ConnectToMaster: received response from server master-127.29.0.188:40013: OK, [479ms] Sub RPC ConnectToMaster: received response from server master-127.29.0.190:45183: OK, [480ms] Sub RPC ConnectToMaster: received response from server master-127.29.0.189:46475: OK, deferred=null) after lookup
14:09:40.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.683214 29814 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34048:
14:09:40.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106"
14:09:40.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
14:09:40.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:09:40.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
14:09:40.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:09:40.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
14:09:40.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:09:40.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:09:40.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:09:40.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:09:40.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:09:40.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:09:40.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
14:09:40.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
14:09:40.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
14:09:40.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
14:09:40.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
14:09:40.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:09:40.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:09:40.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:09:40.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:09:40.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
14:09:40.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
14:09:40.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:09:40.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:09:40.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:09:40.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:40.683570 29814 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
14:09:40.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.694079 30317 tablet_service.cc:1511] Processing CreateTablet for tablet 5b5fca86382a4532a7aaef4dc5ca1391 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106 [id=4e45894abdfb4a83a3a3fbc3b53bec2b]), partition=RANGE (key) PARTITION UNBOUNDED
14:09:40.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.694108 30047 tablet_service.cc:1511] Processing CreateTablet for tablet 5b5fca86382a4532a7aaef4dc5ca1391 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106 [id=4e45894abdfb4a83a3a3fbc3b53bec2b]), partition=RANGE (key) PARTITION UNBOUNDED
14:09:40.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.694394 30047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b5fca86382a4532a7aaef4dc5ca1391. 1 dirs total, 0 dirs full, 0 dirs failed
14:09:40.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.694394 30317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b5fca86382a4532a7aaef4dc5ca1391. 1 dirs total, 0 dirs full, 0 dirs failed
14:09:40.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.696724 30415 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Bootstrap starting.
14:09:40.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.697542 30415 tablet_bootstrap.cc:654] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Neither blocks nor log segments found. Creating new log.
14:09:40.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.697876 30415 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Log is configured to *not* fsync() on all Append() calls
14:09:40.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.697885 30182 tablet_service.cc:1511] Processing CreateTablet for tablet 5b5fca86382a4532a7aaef4dc5ca1391 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106 [id=4e45894abdfb4a83a3a3fbc3b53bec2b]), partition=RANGE (key) PARTITION UNBOUNDED
14:09:40.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.698123 30182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b5fca86382a4532a7aaef4dc5ca1391. 1 dirs total, 0 dirs full, 0 dirs failed
14:09:40.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.698472 30415 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: No bootstrap required, opened a new log
14:09:40.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.698552 30415 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
14:09:40.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.700779 30415 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:40.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.701000 30418 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Bootstrap starting.
14:09:40.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.701303 30415 raft_consensus.cc:385] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:09:40.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.701382 30415 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d90e981114c450186889150658715c8, State: Initialized, Role: FOLLOWER
14:09:40.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.701488 30415 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:40.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.701711 30415 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
14:09:40.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.702188 30416 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap starting.
14:09:40.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.702881 30416 tablet_bootstrap.cc:654] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Neither blocks nor log segments found. Creating new log.
14:09:40.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.703135 30416 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Log is configured to *not* fsync() on all Append() calls
14:09:40.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.708429 30418 tablet_bootstrap.cc:654] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Neither blocks nor log segments found. Creating new log.
14:09:40.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.708736 30418 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Log is configured to *not* fsync() on all Append() calls
14:09:40.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.710821 30418 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: No bootstrap required, opened a new log
14:09:40.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.710917 30418 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.001s
14:09:40.711 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 4e45894abdfb4a83a3a3fbc3b53bec2b
14:09:40.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.712109 30416 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: No bootstrap required, opened a new log
14:09:40.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.712198 30416 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Time spent bootstrapping tablet: real 0.010s	user 0.001s	sys 0.000s
14:09:40.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.712456 30418 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:40.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.712615 30418 raft_consensus.cc:385] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:09:40.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.712647 30418 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f958fcb6937451184fba50df6455e4d, State: Initialized, Role: FOLLOWER
14:09:40.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.712742 30418 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:40.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.712996 30418 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
14:09:40.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.714327 30416 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:40.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.714481 30416 raft_consensus.cc:385] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:09:40.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.714515 30416 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1351807c09ec4b70b1adf098a1d80986, State: Initialized, Role: FOLLOWER
14:09:40.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.715011 30416 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:40.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.715574 30385 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:09:40.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.715718 30416 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
14:09:40.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:40.726224 30116 tablet.cc:2406] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:09:40.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:40.740520 30386 tablet.cc:2406] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:09:40.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.856940 30423 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:09:40.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:40.856971 30251 tablet.cc:2406] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:09:40.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.857093 30423 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:40.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.857456 30423 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:40.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.861616 30067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "1351807c09ec4b70b1adf098a1d80986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d90e981114c450186889150658715c8" is_pre_election: true
14:09:40.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.861795 30067 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1351807c09ec4b70b1adf098a1d80986 in term 0.
14:09:40.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.861845 30202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "1351807c09ec4b70b1adf098a1d80986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d" is_pre_election: true
14:09:40.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.862010 30202 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1351807c09ec4b70b1adf098a1d80986 in term 0.
14:09:40.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.862017 30271 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1351807c09ec4b70b1adf098a1d80986, 2d90e981114c450186889150658715c8; no voters: 
14:09:40.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.862160 30423 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 0 FOLLOWER]: Leader pre-election won for term 1
14:09:40.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.862210 30423 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:09:40.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.862234 30423 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 0 FOLLOWER]: Advancing to term 1
14:09:40.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.941282 30423 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:40.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.941543 30423 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 1 election: Requested vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:40.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.941861 30202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "1351807c09ec4b70b1adf098a1d80986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d"
14:09:40.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.941936 30067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "1351807c09ec4b70b1adf098a1d80986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d90e981114c450186889150658715c8"
14:09:40.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.941987 30202 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 0 FOLLOWER]: Advancing to term 1
14:09:40.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.942057 30067 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 0 FOLLOWER]: Advancing to term 1
14:09:40.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.942790 30067 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1351807c09ec4b70b1adf098a1d80986 in term 1.
14:09:40.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.942999 30202 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1351807c09ec4b70b1adf098a1d80986 in term 1.
14:09:40.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.943032 30271 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1351807c09ec4b70b1adf098a1d80986, 2d90e981114c450186889150658715c8; no voters: 
14:09:40.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.943200 30423 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 1 FOLLOWER]: Leader election won for term 1
14:09:40.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.943364 30423 raft_consensus.cc:697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 1 LEADER]: Becoming Leader. State: Replica: 1351807c09ec4b70b1adf098a1d80986, State: Running, Role: LEADER
14:09:40.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.943456 30423 consensus_queue.cc:237] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:40.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:40.944427 29814 catalog_manager.cc:5697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131). New cstate: current_term: 1 leader_uuid: "1351807c09ec4b70b1adf098a1d80986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } health_report { overall_health: HEALTHY } } }
14:09:40.986 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 4e45894abdfb4a83a3a3fbc3b53bec2b
14:09:41.005 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:41.011 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299993, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:41.013 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [7ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=null) after lookup
14:09:41.025 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:41.027 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] Successfully connected to peer
14:09:41.093 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0x537a267f, L:/127.0.0.1:51698 - R:/127.29.0.131:38303] using TOKEN/null
14:09:41.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:41.095963 30202 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 1 FOLLOWER]: Refusing update from remote peer 1351807c09ec4b70b1adf098a1d80986: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:09:41.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:41.096030 30067 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 1 FOLLOWER]: Refusing update from remote peer 1351807c09ec4b70b1adf098a1d80986: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:09:41.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:41.096377 30426 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:09:41.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:41.096583 30423 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:09:41.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:41.104700 30433 mvcc.cc:204] Tried to move back new op lower bound from 7305297023366291456 to 7305297022745026560. Current Snapshot: MvccSnapshot[applied={T|T < 7305297023366291456}]
14:09:41.168 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:09:41.170 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.0.131:38303
14:09:41.171 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:09:41.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:41.172293 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 30255
14:09:41.178 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:09:41.180 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] handling channelInactive
14:09:41.180 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state READY due to: connection closed
14:09:41.181 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: connection closed
14:09:41.202 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:41.203 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:41.203 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [3ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1930773485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2752b1b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5ae3acc0 -> wakeup thread Time-limited test)) after lookup
14:09:41.204 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:41.206 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.206 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.207 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.223 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:41.223 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:41.224 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [3ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [29ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1930773485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2752b1b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5ae3acc0 -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062))) after lookup
14:09:41.225 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:41.226 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.227 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.227 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.263 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:41.263 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:41.264 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [3ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [29ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [46ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [49ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [49ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1930773485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2752b1b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062) -> (continuation of Deferred@21453116 after retry RPC@1639369013), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5ae3acc0 -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062) -> (continuation of Deferred@21453116 after retry RPC@1639369013))) after lookup
14:09:41.265 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:41.267 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.267 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.267 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.302 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:41.302 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:41.303 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=124), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [3ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [29ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [46ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [49ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [49ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [86ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [89ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [89ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [121ms] refreshing cache from master, [122ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [124ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1930773485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2752b1b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062) -> (continuation of Deferred@21453116 after retry RPC@1639369013) -> (continuation of Deferred@293471042 after retry RPC@280254760), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5ae3acc0 -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062) -> (continuation of Deferred@21453116 after retry RPC@1639369013) -> (continuation of Deferred@293471042 after retry RPC@280254760))) after lookup
14:09:41.304 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:41.306 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.306 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.306 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.503 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:41.503 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:41.504 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=325), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [3ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [29ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [46ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [49ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [49ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [86ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [89ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [89ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [121ms] refreshing cache from master, [122ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [124ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [126ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [128ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [128ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [322ms] refreshing cache from master, [322ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [324ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1930773485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2752b1b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062) -> (continuation of Deferred@21453116 after retry RPC@1639369013) -> (continuation of Deferred@293471042 after retry RPC@280254760) -> (continuation of Deferred@1360551596 after retry RPC@1667251299), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5ae3acc0 -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062) -> (continuation of Deferred@21453116 after retry RPC@1639369013) -> (continuation of Deferred@293471042 after retry RPC@280254760) -> (continuation of Deferred@1360551596 after retry RPC@1667251299))) after lookup
14:09:41.505 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:41.506 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.506 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:41.506 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:42.402 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:42.403 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:42.403 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1224), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [3ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [29ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [46ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [49ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [49ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [86ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [89ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [89ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [121ms] refreshing cache from master, [122ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [124ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [126ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [128ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [128ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [322ms] refreshing cache from master, [322ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [324ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [326ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [328ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [328ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [1222ms] refreshing cache from master, [1222ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [1224ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1930773485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2752b1b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062) -> (continuation of Deferred@21453116 after retry RPC@1639369013) -> (continuation of Deferred@293471042 after retry RPC@280254760) -> (continuation of Deferred@1360551596 after retry RPC@1667251299) -> (continuation of Deferred@980024062 after retry RPC@1485327092), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5ae3acc0 -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062) -> (continuation of Deferred@21453116 after retry RPC@1639369013) -> (continuation of Deferred@293471042 after retry RPC@280254760) -> (continuation of Deferred@1360551596 after retry RPC@1667251299) -> (continuation of Deferred@980024062 after retry RPC@1485327092))) after lookup
14:09:42.404 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:42.405 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:42.405 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:42.405 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:42.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.626823 30445 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1351807c09ec4b70b1adf098a1d80986)
14:09:42.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.626920 30445 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:42.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.626927 30446 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1351807c09ec4b70b1adf098a1d80986)
14:09:42.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.627012 30446 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:42.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.627228 30445 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:42.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:42.627782 30138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:09:42.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:42.628399 30138 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:09:42.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:42.628412 30003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:09:42.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.628576 30446 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:42.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:42.629065 30003 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:09:42.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.630730 30067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "0f958fcb6937451184fba50df6455e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2d90e981114c450186889150658715c8" is_pre_election: true
14:09:42.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.630831 30067 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f958fcb6937451184fba50df6455e4d in term 1.
14:09:42.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.630847 30202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d" is_pre_election: true
14:09:42.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.630924 30202 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d90e981114c450186889150658715c8 in term 1.
14:09:42.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.631105 30136 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 2d90e981114c450186889150658715c8; no voters: 1351807c09ec4b70b1adf098a1d80986
14:09:42.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.631112 30000 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 2d90e981114c450186889150658715c8; no voters: 1351807c09ec4b70b1adf098a1d80986
14:09:42.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.631248 30445 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 1 FOLLOWER]: Leader pre-election won for term 2
14:09:42.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.631280 30445 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1351807c09ec4b70b1adf098a1d80986)
14:09:42.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.631325 30445 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 1 FOLLOWER]: Advancing to term 2
14:09:42.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.631311 30446 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
14:09:42.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.631354 30446 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1351807c09ec4b70b1adf098a1d80986)
14:09:42.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.631379 30446 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 1 FOLLOWER]: Advancing to term 2
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.632236 30445 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.632247 30446 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.632370 30445 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 2 election: Requested vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.632529 30446 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 2 election: Requested vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.632575 30202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d"
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.632640 30202 raft_consensus.cc:2393] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2d90e981114c450186889150658715c8 in current term 2: Already voted for candidate 0f958fcb6937451184fba50df6455e4d in this term.
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:42.632928 30138 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.633033 30067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "0f958fcb6937451184fba50df6455e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2d90e981114c450186889150658715c8"
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.633106 30067 raft_consensus.cc:2393] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0f958fcb6937451184fba50df6455e4d in current term 2: Already voted for candidate 2d90e981114c450186889150658715c8 in this term.
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.633261 30136 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d; no voters: 1351807c09ec4b70b1adf098a1d80986, 2d90e981114c450186889150658715c8
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.633369 30445 raft_consensus.cc:2749] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:42.633739 30003 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:09:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.633790 30003 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d90e981114c450186889150658715c8; no voters: 0f958fcb6937451184fba50df6455e4d, 1351807c09ec4b70b1adf098a1d80986
14:09:42.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:42.633865 30446 raft_consensus.cc:2749] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
14:09:44.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.219713 30456 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:09:44.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.219842 30456 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:44.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.220098 30456 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:44.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.220386 30202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d" is_pre_election: true
14:09:44.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.220510 30202 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d90e981114c450186889150658715c8 in term 2.
14:09:44.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.220695 30000 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 2d90e981114c450186889150658715c8; no voters: 
14:09:44.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.220794 30456 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
14:09:44.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.220830 30456 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:09:44.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.220844 30456 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 2 FOLLOWER]: Advancing to term 3
14:09:44.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:44.221009 30003 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:09:44.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.221448 30456 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:44.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.221602 30456 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 3 election: Requested vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:44.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.221762 30202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d"
14:09:44.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.221845 30202 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 2 FOLLOWER]: Advancing to term 3
14:09:44.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:44.222174 30003 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:09:44.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.222604 30202 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d90e981114c450186889150658715c8 in term 3.
14:09:44.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.222783 30000 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 2d90e981114c450186889150658715c8; no voters: 1351807c09ec4b70b1adf098a1d80986
14:09:44.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.222904 30456 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 3 FOLLOWER]: Leader election won for term 3
14:09:44.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.222990 30456 raft_consensus.cc:697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 3 LEADER]: Becoming Leader. State: Replica: 2d90e981114c450186889150658715c8, State: Running, Role: LEADER
14:09:44.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.223206 30456 consensus_queue.cc:237] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:44.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.223953 29814 catalog_manager.cc:5697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 reported cstate change: term changed from 1 to 3, leader changed from 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131) to 2d90e981114c450186889150658715c8 (127.29.0.129). New cstate: current_term: 3 leader_uuid: "2d90e981114c450186889150658715c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } health_report { overall_health: UNKNOWN } } }
14:09:44.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.712437 30202 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 3 FOLLOWER]: Refusing update from remote peer 2d90e981114c450186889150658715c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
14:09:44.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:44.712808 30456 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
14:09:44.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:44.714847 30003 consensus_peers.cc:597] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 -> Peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Couldn't send request to peer 1351807c09ec4b70b1adf098a1d80986. Status: Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:09:45.582 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:45.582 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:45.583 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=4404), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [3ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [29ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [46ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [49ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [49ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [86ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [89ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [89ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [121ms] refreshing cache from master, [122ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [124ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [126ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [128ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [128ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [322ms] refreshing cache from master, [322ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [324ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [326ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [328ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [328ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [1222ms] refreshing cache from master, [1222ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [1224ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [1226ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [1227ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [1227ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [4401ms] refreshing cache from master, [4401ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [4404ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1930773485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2752b1b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062) -> (continuation of Deferred@21453116 after retry RPC@1639369013) -> (continuation of Deferred@293471042 after retry RPC@280254760) -> (continuation of Deferred@1360551596 after retry RPC@1667251299) -> (continuation of Deferred@980024062 after retry RPC@1485327092) -> (continuation of Deferred@694925494 after retry RPC@132152122), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5ae3acc0 -> wakeup thread Time-limited test -> (continuation of Deferred@1624268726 after retry RPC@164611062) -> (continuation of Deferred@21453116 after retry RPC@1639369013) -> (continuation of Deferred@293471042 after retry RPC@280254760) -> (continuation of Deferred@1360551596 after retry RPC@1667251299) -> (continuation of Deferred@980024062 after retry RPC@1485327092) -> (continuation of Deferred@694925494 after retry RPC@132152122))) after lookup
14:09:45.584 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:45.585 [DEBUG - kudu-nio-2] (Connection.java:757) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] Successfully connected to peer
14:09:45.608 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0x69740a5d, L:/127.0.0.1:59738 - R:/127.29.0.129:37851] using TOKEN/null
14:09:45.619 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.0.131:38303
14:09:45.620 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:09:45.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.621405 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:45.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:45.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/logs
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:45.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.131:38303
14:09:45.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.131
14:09:45.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:45.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40737
14:09:45.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.131
14:09:45.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:45.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:45.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:45.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:09:45.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:45.699149 30485 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:45.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:45.699326 30485 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:45.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:45.699348 30485 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:45.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:45.700893 30485 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:45.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:45.700953 30485 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.131
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.702370 30485 tablet_server_runner.cc:78] Tablet server non-default flags:
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.131:38303
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:45.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.131
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40737
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.30485
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.131
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/logs
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.702557 30485 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:45.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.702737 30485 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:45.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:45.705539 30493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:45.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:45.705550 30490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:45.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.705554 30485 server_base.cc:1061] running on GCE node
14:09:45.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:45.705554 30491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:45.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.705924 30485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:45.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.706112 30485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:45.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.707192 30485 hybrid_clock.cc:648] HybridClock initialized: now 1783519785707172 us; error 34 us; skew 500 ppm
14:09:45.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.708777 30485 webserver.cc:533] Webserver started at http://127.29.0.131:40737/ using document root <none> and password file <none>
14:09:45.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.709035 30485 fs_manager.cc:362] Metadata directory not provided
14:09:45.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.709120 30485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:45.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.710880 30485 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.002s	sys 0.000s
14:09:45.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.711596 30499 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:45.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.711812 30485 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
14:09:45.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.711879 30485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:45.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1351807c09ec4b70b1adf098a1d80986"
14:09:45.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:45.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.712200 30485 fs_report.cc:389] FS layout report
14:09:45.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:45.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:45.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:45.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/data
14:09:45.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:45.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:45.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:45.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:45.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:45.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:45.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:45.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:45.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:45.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:45.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:09:45.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.722440 30485 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:09:45.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.722692 30485 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:09:45.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.722805 30485 kserver.cc:163] Server-wide thread pool size limit: 3276
14:09:45.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.723091 30485 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:09:45.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.723555 30506 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:09:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.724483 30485 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:09:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.724532 30485 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:09:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.724557 30485 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:09:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.725044 30485 ts_tablet_manager.cc:616] Registered 1 tablets
14:09:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.725072 30485 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:09:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.725183 30506 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap starting.
14:09:45.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.732867 30485 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.131:38303
14:09:45.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.733299 30485 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:09:45.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.732944 30613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.131:38303 every 8 connection(s)
14:09:45.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.734894 30506 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Log is configured to *not* fsync() on all Append() calls
14:09:45.736 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:09:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.736289 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 30485
14:09:45.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.737200 30506 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:09:45.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.737471 30506 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap complete.
14:09:45.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.739341 30506 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.007s
14:09:45.741 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:09:45.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.740790 30506 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:45.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.741305 30506 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1351807c09ec4b70b1adf098a1d80986, State: Initialized, Role: FOLLOWER
14:09:45.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.741461 30506 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:45.741 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.0.129:37851
14:09:45.742 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:09:45.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.741683 30506 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
14:09:45.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.742409 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 29985
14:09:45.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.744704 30615 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:09:45.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.744808 30615 heartbeater.cc:461] Registering TS with master...
14:09:45.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.745011 30615 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:09:45.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.745673 29741 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:45.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.747792 30614 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:09:45.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.747870 30614 heartbeater.cc:461] Registering TS with master...
14:09:45.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.747972 30614 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:09:45.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.748181 30616 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:09:45.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.748234 30616 heartbeater.cc:461] Registering TS with master...
14:09:45.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.748325 30616 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:09:45.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.748564 29886 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:45.749 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:09:45.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.749078 29814 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:45.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.749545 29814 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.131:56829
14:09:45.750 [DEBUG - kudu-nio-2] (Connection.java:239) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] handling channelInactive
14:09:45.750 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state READY due to: connection closed
14:09:45.750 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: connection closed
14:09:45.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:45.750161 30616 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:09:45.763 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:45.764 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:45.764 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [1ms] received response from server 2d90e981114c450186889150658715c8: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@626731759(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2ca91e23 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1cb9afbc -> wakeup thread Time-limited test)) after lookup
14:09:45.765 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:45.766 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.766 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.766 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.783 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:45.783 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:45.784 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [1ms] received response from server 2d90e981114c450186889150658715c8: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 2d90e981114c450186889150658715c8, [17ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [18ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@626731759(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2ca91e23 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1cb9afbc -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553))) after lookup
14:09:45.784 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:45.785 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.786 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.786 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.822 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:45.822 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:45.823 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [1ms] received response from server 2d90e981114c450186889150658715c8: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 2d90e981114c450186889150658715c8, [17ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [18ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [35ms] sending RPC to server 2d90e981114c450186889150658715c8, [37ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [37ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [73ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@626731759(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2ca91e23 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553) -> (continuation of Deferred@279252680 after retry RPC@1096176161), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1cb9afbc -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553) -> (continuation of Deferred@279252680 after retry RPC@1096176161))) after lookup
14:09:45.824 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:45.825 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.826 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.826 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.862 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:45.863 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:45.863 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=113), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [1ms] received response from server 2d90e981114c450186889150658715c8: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 2d90e981114c450186889150658715c8, [17ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [18ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [35ms] sending RPC to server 2d90e981114c450186889150658715c8, [37ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [37ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [73ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [75ms] sending RPC to server 2d90e981114c450186889150658715c8, [77ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [77ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [110ms] refreshing cache from master, [110ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@626731759(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2ca91e23 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553) -> (continuation of Deferred@279252680 after retry RPC@1096176161) -> (continuation of Deferred@139908777 after retry RPC@2078427864), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1cb9afbc -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553) -> (continuation of Deferred@279252680 after retry RPC@1096176161) -> (continuation of Deferred@139908777 after retry RPC@2078427864))) after lookup
14:09:45.864 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:45.865 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.865 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.866 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.942 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:45.942 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:45.942 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=192), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [1ms] received response from server 2d90e981114c450186889150658715c8: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 2d90e981114c450186889150658715c8, [17ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [18ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [35ms] sending RPC to server 2d90e981114c450186889150658715c8, [37ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [37ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [73ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [75ms] sending RPC to server 2d90e981114c450186889150658715c8, [77ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [77ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [110ms] refreshing cache from master, [110ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [115ms] sending RPC to server 2d90e981114c450186889150658715c8, [117ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [117ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [190ms] refreshing cache from master, [190ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [192ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@626731759(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2ca91e23 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553) -> (continuation of Deferred@279252680 after retry RPC@1096176161) -> (continuation of Deferred@139908777 after retry RPC@2078427864) -> (continuation of Deferred@2001819674 after retry RPC@512518306), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1cb9afbc -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553) -> (continuation of Deferred@279252680 after retry RPC@1096176161) -> (continuation of Deferred@139908777 after retry RPC@2078427864) -> (continuation of Deferred@2001819674 after retry RPC@512518306))) after lookup
14:09:45.943 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:45.944 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.944 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:45.945 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:46.642 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:46.642 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:46.643 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=893), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [1ms] received response from server 2d90e981114c450186889150658715c8: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 2d90e981114c450186889150658715c8, [17ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [18ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [35ms] sending RPC to server 2d90e981114c450186889150658715c8, [37ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [37ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [73ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [75ms] sending RPC to server 2d90e981114c450186889150658715c8, [77ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [77ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [110ms] refreshing cache from master, [110ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [115ms] sending RPC to server 2d90e981114c450186889150658715c8, [117ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [117ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [190ms] refreshing cache from master, [190ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [192ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [194ms] sending RPC to server 2d90e981114c450186889150658715c8, [196ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [196ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [890ms] refreshing cache from master, [891ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [893ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@626731759(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2ca91e23 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553) -> (continuation of Deferred@279252680 after retry RPC@1096176161) -> (continuation of Deferred@139908777 after retry RPC@2078427864) -> (continuation of Deferred@2001819674 after retry RPC@512518306) -> (continuation of Deferred@727842418 after retry RPC@1073587402), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1cb9afbc -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553) -> (continuation of Deferred@279252680 after retry RPC@1096176161) -> (continuation of Deferred@139908777 after retry RPC@2078427864) -> (continuation of Deferred@2001819674 after retry RPC@512518306) -> (continuation of Deferred@727842418 after retry RPC@1073587402))) after lookup
14:09:46.643 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:46.645 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:46.645 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:46.645 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:47.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.118486 30627 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d90e981114c450186889150658715c8)
14:09:47.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.118604 30627 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:47.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.118779 30627 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:47.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:47.119577 30136 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2d90e981114c450186889150658715c8 (127.29.0.129:37851): Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111)
14:09:47.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.122026 30568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "0f958fcb6937451184fba50df6455e4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "1351807c09ec4b70b1adf098a1d80986" is_pre_election: true
14:09:47.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.122156 30568 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f958fcb6937451184fba50df6455e4d in term 1.
14:09:47.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.122437 30138 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 1351807c09ec4b70b1adf098a1d80986; no voters: 2d90e981114c450186889150658715c8
14:09:47.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.122582 30627 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 3 FOLLOWER]: Leader pre-election won for term 4
14:09:47.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.122653 30627 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2d90e981114c450186889150658715c8)
14:09:47.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.122677 30627 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 3 FOLLOWER]: Advancing to term 4
14:09:47.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.123314 30627 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:47.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.123456 30627 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 4 election: Requested vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:47.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.123755 30568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "0f958fcb6937451184fba50df6455e4d" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "1351807c09ec4b70b1adf098a1d80986"
14:09:47.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.123850 30568 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 1 FOLLOWER]: Advancing to term 4
14:09:47.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:47.124039 30136 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 2d90e981114c450186889150658715c8 (127.29.0.129:37851): Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111)
14:09:47.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.124943 30568 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f958fcb6937451184fba50df6455e4d in term 4.
14:09:47.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.125140 30138 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 1351807c09ec4b70b1adf098a1d80986; no voters: 2d90e981114c450186889150658715c8
14:09:47.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.125238 30627 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 4 FOLLOWER]: Leader election won for term 4
14:09:47.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.125361 30627 raft_consensus.cc:697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 4 LEADER]: Becoming Leader. State: Replica: 0f958fcb6937451184fba50df6455e4d, State: Running, Role: LEADER
14:09:47.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.125420 30627 consensus_queue.cc:237] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:47.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.126099 29814 catalog_manager.cc:5697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d reported cstate change: term changed from 3 to 4, leader changed from 2d90e981114c450186889150658715c8 (127.29.0.129) to 0f958fcb6937451184fba50df6455e4d (127.29.0.130). New cstate: current_term: 4 leader_uuid: "0f958fcb6937451184fba50df6455e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } health_report { overall_health: UNKNOWN } } }
14:09:47.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.626982 30568 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 4 FOLLOWER]: Refusing update from remote peer 0f958fcb6937451184fba50df6455e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
14:09:47.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.627332 30644 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
14:09:47.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:47.629089 30646 mvcc.cc:204] Tried to move back new op lower bound from 7305297041886449664 to 7305297036178862080. Current Snapshot: MvccSnapshot[applied={T|T < 7305297041872322560}]
14:09:47.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:47.630460 30136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111) [suppressed 3 similar messages]
14:09:47.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:47.630941 30136 consensus_peers.cc:597] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d -> Peer 2d90e981114c450186889150658715c8 (127.29.0.129:37851): Couldn't send request to peer 2d90e981114c450186889150658715c8. Status: Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:09:50.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:50.022553 30136 consensus_peers.cc:597] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d -> Peer 2d90e981114c450186889150658715c8 (127.29.0.129:37851): Couldn't send request to peer 2d90e981114c450186889150658715c8. Status: Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:09:50.603 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:50.603 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:50.604 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=4854), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [1ms] received response from server 2d90e981114c450186889150658715c8: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 2d90e981114c450186889150658715c8, [17ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [18ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [35ms] sending RPC to server 2d90e981114c450186889150658715c8, [37ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [37ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [73ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [75ms] sending RPC to server 2d90e981114c450186889150658715c8, [77ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [77ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [110ms] refreshing cache from master, [110ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [115ms] sending RPC to server 2d90e981114c450186889150658715c8, [117ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [117ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [190ms] refreshing cache from master, [190ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [192ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [194ms] sending RPC to server 2d90e981114c450186889150658715c8, [196ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [196ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [890ms] refreshing cache from master, [891ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [893ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [894ms] sending RPC to server 2d90e981114c450186889150658715c8, [896ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [896ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [4850ms] refreshing cache from master, [4851ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [4853ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@626731759(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@2ca91e23 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553) -> (continuation of Deferred@279252680 after retry RPC@1096176161) -> (continuation of Deferred@139908777 after retry RPC@2078427864) -> (continuation of Deferred@2001819674 after retry RPC@512518306) -> (continuation of Deferred@727842418 after retry RPC@1073587402) -> (continuation of Deferred@1139167447 after retry RPC@300906389), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1cb9afbc -> wakeup thread Time-limited test -> (continuation of Deferred@320335232 after retry RPC@1115841553) -> (continuation of Deferred@279252680 after retry RPC@1096176161) -> (continuation of Deferred@139908777 after retry RPC@2078427864) -> (continuation of Deferred@2001819674 after retry RPC@512518306) -> (continuation of Deferred@727842418 after retry RPC@1073587402) -> (continuation of Deferred@1139167447 after retry RPC@300906389))) after lookup
14:09:50.605 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:50.606 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] Successfully connected to peer
14:09:50.623 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xf329ed0d, L:/127.0.0.1:37274 - R:/127.29.0.130:34547] using TOKEN/null
14:09:50.636 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.0.129:37851
14:09:50.637 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.637975 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/logs
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:09:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.129:37851
14:09:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.129
14:09:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44997
14:09:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.129
14:09:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:09:50.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:50.727269 30669 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:50.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:50.727476 30669 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:50.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:50.727497 30669 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:50.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:50.729034 30669 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:50.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:50.729092 30669 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.129
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.730536 30669 tablet_server_runner.cc:78] Tablet server non-default flags:
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.129:37851
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.129
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44997
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.30669
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.129
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/logs
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:50.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.730702 30669 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:50.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.730872 30669 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:50.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:50.733353 30675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:50.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:50.733592 30674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:50.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:50.733781 30677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:50.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.734408 30669 server_base.cc:1061] running on GCE node
14:09:50.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.734764 30669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:50.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.735001 30669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:50.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.736186 30669 hybrid_clock.cc:648] HybridClock initialized: now 1783519790736166 us; error 35 us; skew 500 ppm
14:09:50.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.737367 30669 webserver.cc:533] Webserver started at http://127.29.0.129:44997/ using document root <none> and password file <none>
14:09:50.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.737617 30669 fs_manager.cc:362] Metadata directory not provided
14:09:50.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.737684 30669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:50.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.738945 30669 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.000s	sys 0.000s
14:09:50.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.739751 30683 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.740160 30669 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
14:09:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.740243 30669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2d90e981114c450186889150658715c8"
14:09:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.740481 30669 fs_report.cc:389] FS layout report
14:09:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/data
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:09:50.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.765228 30669 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:09:50.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.765549 30669 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:09:50.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.765669 30669 kserver.cc:163] Server-wide thread pool size limit: 3276
14:09:50.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.765908 30669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:09:50.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.766412 30690 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:09:50.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.767354 30669 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:09:50.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.767432 30669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:09:50.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.767469 30669 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:09:50.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.768165 30669 ts_tablet_manager.cc:616] Registered 1 tablets
14:09:50.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.768220 30669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:09:50.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.768241 30690 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Bootstrap starting.
14:09:50.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.776189 30669 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.129:37851
14:09:50.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.776614 30797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.129:37851 every 8 connection(s)
14:09:50.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.776656 30669 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:09:50.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.777827 30690 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Log is configured to *not* fsync() on all Append() calls
14:09:50.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.780892 30690 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:09:50.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.781225 30690 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Bootstrap complete.
14:09:50.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.782884 30690 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.008s
14:09:50.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.783844 30690 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:50.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.784050 30690 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d90e981114c450186889150658715c8, State: Initialized, Role: FOLLOWER
14:09:50.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.784168 30690 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:50.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.784400 30690 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
14:09:50.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.785941 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 30669
14:09:50.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.786113 30798 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:09:50.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.786211 30798 heartbeater.cc:461] Registering TS with master...
14:09:50.786 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:09:50.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.786408 30798 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:09:50.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.788107 30799 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:09:50.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.788208 30799 heartbeater.cc:461] Registering TS with master...
14:09:50.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.788316 30799 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:09:50.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.788755 29741 ts_manager.cc:194] Re-registered known tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:09:50.791 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:09:50.791 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.0.130:34547
14:09:50.792 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:09:50.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.790174 30800 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:09:50.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.790241 30800 heartbeater.cc:461] Registering TS with master...
14:09:50.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.790271 29887 ts_manager.cc:194] Re-registered known tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:09:50.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.790341 30800 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:09:50.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.791878 29813 ts_manager.cc:194] Re-registered known tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:09:50.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.792240 29813 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.129:39321
14:09:50.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.792564 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 30120
14:09:50.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:50.793012 30800 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:09:50.797 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:09:50.798 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] handling channelInactive
14:09:50.798 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state READY due to: connection closed
14:09:50.799 [DEBUG - Time-limited test] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:50.800 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.800 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.800 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.822 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:50.823 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:50.823 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1459855793(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@68bb19ef -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@713a85bd -> wakeup thread Time-limited test)) after lookup
14:09:50.824 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:50.825 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.825 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.825 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.842 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:50.842 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:50.843 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [27ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [27ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1459855793(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@68bb19ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@713a85bd -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275))) after lookup
14:09:50.844 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:50.845 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.845 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.845 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.883 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:50.883 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:50.883 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [27ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [27ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [46ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [47ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [47ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1459855793(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@68bb19ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275) -> (continuation of Deferred@1584052885 after retry RPC@2102068533), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@713a85bd -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275) -> (continuation of Deferred@1584052885 after retry RPC@2102068533))) after lookup
14:09:50.884 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:50.885 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.885 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.885 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.961 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:50.962 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:50.962 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=164), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [27ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [27ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [46ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [47ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [47ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [85ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [87ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [87ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [161ms] refreshing cache from master, [161ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [163ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1459855793(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@68bb19ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275) -> (continuation of Deferred@1584052885 after retry RPC@2102068533) -> (continuation of Deferred@769373551 after retry RPC@144469836), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@713a85bd -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275) -> (continuation of Deferred@1584052885 after retry RPC@2102068533) -> (continuation of Deferred@769373551 after retry RPC@144469836))) after lookup
14:09:50.963 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:50.964 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.964 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:50.964 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:51.182 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:51.182 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:51.182 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=384), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [27ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [27ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [46ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [47ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [47ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [85ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [87ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [87ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [161ms] refreshing cache from master, [161ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [163ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [164ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [166ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [166ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [381ms] refreshing cache from master, [381ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [383ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1459855793(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@68bb19ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275) -> (continuation of Deferred@1584052885 after retry RPC@2102068533) -> (continuation of Deferred@769373551 after retry RPC@144469836) -> (continuation of Deferred@1786805286 after retry RPC@17668149), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@713a85bd -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275) -> (continuation of Deferred@1584052885 after retry RPC@2102068533) -> (continuation of Deferred@769373551 after retry RPC@144469836) -> (continuation of Deferred@1786805286 after retry RPC@17668149))) after lookup
14:09:51.183 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:51.184 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:51.184 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:51.184 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:51.802 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:51.803 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:51.803 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1005), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [27ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [27ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [46ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [47ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [47ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [85ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [87ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [87ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [161ms] refreshing cache from master, [161ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [163ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [164ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [166ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [166ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [381ms] refreshing cache from master, [381ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [383ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [385ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [386ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [386ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [1002ms] refreshing cache from master, [1002ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [1004ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1459855793(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@68bb19ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275) -> (continuation of Deferred@1584052885 after retry RPC@2102068533) -> (continuation of Deferred@769373551 after retry RPC@144469836) -> (continuation of Deferred@1786805286 after retry RPC@17668149) -> (continuation of Deferred@861361310 after retry RPC@1593229041), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@713a85bd -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275) -> (continuation of Deferred@1584052885 after retry RPC@2102068533) -> (continuation of Deferred@769373551 after retry RPC@144469836) -> (continuation of Deferred@1786805286 after retry RPC@17668149) -> (continuation of Deferred@861361310 after retry RPC@1593229041))) after lookup
14:09:51.803 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:51.804 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:51.804 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:51.805 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:51.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:51.933254 30811 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:09:51.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:51.933413 30811 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:51.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:51.933749 30811 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:51.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:51.934180 30684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:09:51.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:51.934711 30684 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547): Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:09:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:51.936931 30568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "1351807c09ec4b70b1adf098a1d80986" is_pre_election: true
14:09:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:51.937213 30687 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d90e981114c450186889150658715c8; no voters: 0f958fcb6937451184fba50df6455e4d, 1351807c09ec4b70b1adf098a1d80986
14:09:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:51.937351 30811 raft_consensus.cc:2749] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
14:09:52.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.135908 30816 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0f958fcb6937451184fba50df6455e4d)
14:09:52.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.136022 30816 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:52.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.136375 30816 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:52.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:52.136868 30500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:09:52.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:52.137351 30500 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547): Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:09:52.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.139519 30752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "1351807c09ec4b70b1adf098a1d80986" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "2d90e981114c450186889150658715c8" is_pre_election: true
14:09:52.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.139721 30752 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1351807c09ec4b70b1adf098a1d80986 in term 3.
14:09:52.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.140002 30501 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1351807c09ec4b70b1adf098a1d80986, 2d90e981114c450186889150658715c8; no voters: 0f958fcb6937451184fba50df6455e4d
14:09:52.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.140141 30816 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 4 FOLLOWER]: Leader pre-election won for term 5
14:09:52.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.140196 30816 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0f958fcb6937451184fba50df6455e4d)
14:09:52.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.140213 30816 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 4 FOLLOWER]: Advancing to term 5
14:09:52.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.140776 30816 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:52.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.140930 30816 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 5 election: Requested vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:52.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.141094 30752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "1351807c09ec4b70b1adf098a1d80986" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "2d90e981114c450186889150658715c8"
14:09:52.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.141178 30752 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 3 FOLLOWER]: Advancing to term 5
14:09:52.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:52.141556 30500 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547): Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:09:52.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.142241 30752 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1351807c09ec4b70b1adf098a1d80986 in term 5.
14:09:52.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.142433 30501 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1351807c09ec4b70b1adf098a1d80986, 2d90e981114c450186889150658715c8; no voters: 0f958fcb6937451184fba50df6455e4d
14:09:52.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.142514 30816 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 5 FOLLOWER]: Leader election won for term 5
14:09:52.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.142640 30816 raft_consensus.cc:697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 5 LEADER]: Becoming Leader. State: Replica: 1351807c09ec4b70b1adf098a1d80986, State: Running, Role: LEADER
14:09:52.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.142720 30816 consensus_queue.cc:237] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:52.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.143435 29813 catalog_manager.cc:5697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 reported cstate change: term changed from 4 to 5, leader changed from 0f958fcb6937451184fba50df6455e4d (127.29.0.130) to 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131). New cstate: current_term: 5 leader_uuid: "1351807c09ec4b70b1adf098a1d80986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } health_report { overall_health: HEALTHY } } }
14:09:52.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.630223 30752 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 5 FOLLOWER]: Refusing update from remote peer 1351807c09ec4b70b1adf098a1d80986: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 13. (index mismatch)
14:09:52.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:52.630582 30816 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
14:09:52.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:52.633231 30500 consensus_peers.cc:597] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 -> Peer 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547): Couldn't send request to peer 0f958fcb6937451184fba50df6455e4d. Status: Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:09:53.042 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:53.042 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:53.043 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=14, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=2244), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [25ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [27ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [27ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [44ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [46ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [47ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [47ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [85ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [87ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [87ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [161ms] refreshing cache from master, [161ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [163ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [164ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [166ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [166ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [381ms] refreshing cache from master, [381ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [383ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [385ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [386ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [386ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [1002ms] refreshing cache from master, [1002ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [1004ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [1005ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [1007ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [1007ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2242ms] refreshing cache from master, [2242ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [2244ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1459855793(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@68bb19ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275) -> (continuation of Deferred@1584052885 after retry RPC@2102068533) -> (continuation of Deferred@769373551 after retry RPC@144469836) -> (continuation of Deferred@1786805286 after retry RPC@17668149) -> (continuation of Deferred@861361310 after retry RPC@1593229041) -> (continuation of Deferred@481203160 after retry RPC@2009380746), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@713a85bd -> wakeup thread Time-limited test -> (continuation of Deferred@88212531 after retry RPC@544243275) -> (continuation of Deferred@1584052885 after retry RPC@2102068533) -> (continuation of Deferred@769373551 after retry RPC@144469836) -> (continuation of Deferred@1786805286 after retry RPC@17668149) -> (continuation of Deferred@861361310 after retry RPC@1593229041) -> (continuation of Deferred@481203160 after retry RPC@2009380746))) after lookup
14:09:53.043 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:53.044 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] Successfully connected to peer
14:09:53.058 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xb84065ff, L:/127.0.0.1:46918 - R:/127.29.0.131:38303] using TOKEN/null
14:09:53.068 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.0.130:34547
14:09:53.068 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:09:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.069290 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data
14:09:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/logs
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.130:34547
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.130
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33025
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.130
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:53.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:09:53.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:53.146461 30844 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:53.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:53.146660 30844 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:53.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:53.146692 30844 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:53.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:53.148456 30844 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:53.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:53.148545 30844 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.130
14:09:53.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.150149 30844 tablet_server_runner.cc:78] Tablet server non-default flags:
14:09:53.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:53.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:53.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:53.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:53.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data
14:09:53.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:53.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:53.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:53.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.130:34547
14:09:53.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:53.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.130
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33025
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.30844
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.130
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/logs
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:53.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:53.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.150398 30844 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:53.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.150657 30844 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:53.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:53.153515 30850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:53.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:53.153554 30852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:53.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.153573 30844 server_base.cc:1061] running on GCE node
14:09:53.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:53.153573 30849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:53.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.154002 30844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:53.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.154241 30844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:53.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.155427 30844 hybrid_clock.cc:648] HybridClock initialized: now 1783519793155408 us; error 41 us; skew 500 ppm
14:09:53.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.156558 30844 webserver.cc:533] Webserver started at http://127.29.0.130:33025/ using document root <none> and password file <none>
14:09:53.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.156777 30844 fs_manager.cc:362] Metadata directory not provided
14:09:53.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.156884 30844 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:53.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.158360 30844 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.000s	sys 0.002s
14:09:53.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.159082 30858 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:53.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.159262 30844 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
14:09:53.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.159389 30844 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0f958fcb6937451184fba50df6455e4d"
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.159646 30844 fs_report.cc:389] FS layout report
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/data
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:09:53.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.167320 30844 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:09:53.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.167531 30844 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:09:53.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.167622 30844 kserver.cc:163] Server-wide thread pool size limit: 3276
14:09:53.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.167824 30844 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:09:53.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.168212 30865 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:09:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.169138 30844 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:09:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.169188 30844 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:09:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.169214 30844 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:09:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.169731 30844 ts_tablet_manager.cc:616] Registered 1 tablets
14:09:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.169761 30844 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:09:53.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.169862 30865 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Bootstrap starting.
14:09:53.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.177057 30844 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.130:34547
14:09:53.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.177100 30972 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.130:34547 every 8 connection(s)
14:09:53.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.177426 30844 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:09:53.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.180210 30865 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Log is configured to *not* fsync() on all Append() calls
14:09:53.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.183346 30973 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:09:53.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.183444 30973 heartbeater.cc:461] Registering TS with master...
14:09:53.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.183629 30865 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:09:53.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.183691 30973 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:09:53.185 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:09:53.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.183965 30865 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Bootstrap complete.
14:09:53.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.185020 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 30844
14:09:53.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.185055 30865 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Time spent bootstrapping tablet: real 0.015s	user 0.006s	sys 0.008s
14:09:53.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.186154 30865 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.186384 30865 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f958fcb6937451184fba50df6455e4d, State: Initialized, Role: FOLLOWER
14:09:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.186436 29886 ts_manager.cc:194] Re-registered known tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.186530 30865 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.186790 30865 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
14:09:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.186916 30975 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:09:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.186971 30975 heartbeater.cc:461] Registering TS with master...
14:09:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.187064 30975 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:09:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.187913 29813 ts_manager.cc:194] Re-registered known tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.188383 29813 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.130:53023
14:09:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.188400 30974 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:09:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.188443 30974 heartbeater.cc:461] Registering TS with master...
14:09:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.188527 30974 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:09:53.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.189297 29741 ts_manager.cc:194] Re-registered known tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:53.189 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:09:53.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.190006 30975 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:09:53.204 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.0.131:38303
14:09:53.204 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:09:53.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:53.204603 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 30485
14:09:53.211 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:09:53.211 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] handling channelInactive
14:09:53.211 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state READY due to: connection closed
14:09:53.212 [DEBUG - Time-limited test] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:53.213 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.213 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.213 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.222 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:53.222 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:53.222 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [2ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@151406085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@427b8ca5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25c68498 -> wakeup thread Time-limited test)) after lookup
14:09:53.223 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:53.224 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.225 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.225 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.242 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:53.243 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:53.243 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [2ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [12ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [14ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@151406085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@427b8ca5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25c68498 -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246))) after lookup
14:09:53.244 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:53.245 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.245 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.246 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.262 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:53.262 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:53.263 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=51), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [2ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [12ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [14ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [33ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [35ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [35ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [51ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@151406085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@427b8ca5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246) -> (continuation of Deferred@1440020107 after retry RPC@38305133), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25c68498 -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246) -> (continuation of Deferred@1440020107 after retry RPC@38305133))) after lookup
14:09:53.264 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:53.265 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.265 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.265 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.402 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:53.402 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:53.402 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=191), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [2ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [12ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [14ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [33ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [35ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [35ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [51ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [52ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [54ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [54ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [190ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@151406085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@427b8ca5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246) -> (continuation of Deferred@1440020107 after retry RPC@38305133) -> (continuation of Deferred@574131770 after retry RPC@1666857989), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25c68498 -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246) -> (continuation of Deferred@1440020107 after retry RPC@38305133) -> (continuation of Deferred@574131770 after retry RPC@1666857989))) after lookup
14:09:53.403 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:53.404 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.404 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.405 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.443 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:53.443 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:53.443 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=232), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [2ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [12ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [14ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [33ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [35ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [35ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [51ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [52ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [54ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [54ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [190ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [192ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [194ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [194ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [229ms] refreshing cache from master, [229ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [231ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@151406085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@427b8ca5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246) -> (continuation of Deferred@1440020107 after retry RPC@38305133) -> (continuation of Deferred@574131770 after retry RPC@1666857989) -> (continuation of Deferred@2057701501 after retry RPC@1211021624), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25c68498 -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246) -> (continuation of Deferred@1440020107 after retry RPC@38305133) -> (continuation of Deferred@574131770 after retry RPC@1666857989) -> (continuation of Deferred@2057701501 after retry RPC@1211021624))) after lookup
14:09:53.444 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:53.446 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.446 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:53.446 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:54.082 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:54.082 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:54.083 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=871), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [2ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [12ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [14ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [33ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [35ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [35ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [51ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [52ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [54ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [54ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [190ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [192ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [194ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [194ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [229ms] refreshing cache from master, [229ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [231ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [233ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [235ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [235ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [869ms] refreshing cache from master, [869ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [871ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@151406085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@427b8ca5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246) -> (continuation of Deferred@1440020107 after retry RPC@38305133) -> (continuation of Deferred@574131770 after retry RPC@1666857989) -> (continuation of Deferred@2057701501 after retry RPC@1211021624) -> (continuation of Deferred@1135008378 after retry RPC@2130463823), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25c68498 -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246) -> (continuation of Deferred@1440020107 after retry RPC@38305133) -> (continuation of Deferred@574131770 after retry RPC@1666857989) -> (continuation of Deferred@2057701501 after retry RPC@1211021624) -> (continuation of Deferred@1135008378 after retry RPC@2130463823))) after lookup
14:09:54.084 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:54.085 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:54.085 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:54.085 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:09:54.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.567770 30986 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1351807c09ec4b70b1adf098a1d80986)
14:09:54.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.567863 30986 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:54.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.568019 30986 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:54.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:54.568811 30687 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:09:54.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.571204 30927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d" is_pre_election: true
14:09:54.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.571421 30927 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d90e981114c450186889150658715c8 in term 4.
14:09:54.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.571810 30684 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 2d90e981114c450186889150658715c8; no voters: 1351807c09ec4b70b1adf098a1d80986
14:09:54.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.571928 30986 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
14:09:54.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.572000 30986 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1351807c09ec4b70b1adf098a1d80986)
14:09:54.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.572026 30986 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 5 FOLLOWER]: Advancing to term 6
14:09:54.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.572772 30986 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:54.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.572913 30986 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 6 election: Requested vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:54.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.573087 30927 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d"
14:09:54.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.573189 30927 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 4 FOLLOWER]: Advancing to term 6
14:09:54.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:54.573350 30687 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:09:54.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.574214 30927 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d90e981114c450186889150658715c8 in term 6.
14:09:54.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.574406 30684 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 2d90e981114c450186889150658715c8; no voters: 1351807c09ec4b70b1adf098a1d80986
14:09:54.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.574514 30986 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 6 FOLLOWER]: Leader election won for term 6
14:09:54.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.574645 30986 raft_consensus.cc:697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 6 LEADER]: Becoming Leader. State: Replica: 2d90e981114c450186889150658715c8, State: Running, Role: LEADER
14:09:54.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.574738 30986 consensus_queue.cc:237] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:54.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:54.575400 29813 catalog_manager.cc:5697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 reported cstate change: term changed from 5 to 6, leader changed from 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131) to 2d90e981114c450186889150658715c8 (127.29.0.129). New cstate: current_term: 6 leader_uuid: "2d90e981114c450186889150658715c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } health_report { overall_health: UNKNOWN } } }
14:09:55.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:55.061626 30687 consensus_peers.cc:597] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 -> Peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Couldn't send request to peer 1351807c09ec4b70b1adf098a1d80986. Status: Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:09:55.082 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:55.082 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:55.082 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1871), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [2ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [12ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [14ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [14ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [33ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [35ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [35ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [51ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [52ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [54ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [54ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [190ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [192ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [194ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [194ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [229ms] refreshing cache from master, [229ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [231ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [233ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [235ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [235ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [869ms] refreshing cache from master, [869ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [871ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [872ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [874ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [874ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [1868ms] refreshing cache from master, [1868ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [1870ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@151406085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@427b8ca5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246) -> (continuation of Deferred@1440020107 after retry RPC@38305133) -> (continuation of Deferred@574131770 after retry RPC@1666857989) -> (continuation of Deferred@2057701501 after retry RPC@1211021624) -> (continuation of Deferred@1135008378 after retry RPC@2130463823) -> (continuation of Deferred@378527967 after retry RPC@168642677), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25c68498 -> wakeup thread Time-limited test -> (continuation of Deferred@1824190725 after retry RPC@2101959246) -> (continuation of Deferred@1440020107 after retry RPC@38305133) -> (continuation of Deferred@574131770 after retry RPC@1666857989) -> (continuation of Deferred@2057701501 after retry RPC@1211021624) -> (continuation of Deferred@1135008378 after retry RPC@2130463823) -> (continuation of Deferred@378527967 after retry RPC@168642677))) after lookup
14:09:55.083 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:55.084 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] Successfully connected to peer
14:09:55.105 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x9df00151, L:/127.0.0.1:56918 - R:/127.29.0.129:37851] using TOKEN/null
14:09:55.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.108506 30927 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 6 FOLLOWER]: Refusing update from remote peer 2d90e981114c450186889150658715c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 12. Preceding OpId from leader: term: 6 index: 18. (index mismatch)
14:09:55.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.108989 30986 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
14:09:55.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.112498 31008 mvcc.cc:204] Tried to move back new op lower bound from 7305297072400265216 to 7305297068617093120. Current Snapshot: MvccSnapshot[applied={T|T < 7305297072373821440}]
14:09:55.122 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.0.131:38303
14:09:55.123 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:09:55.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.123607 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:55.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:55.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:55.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data
14:09:55.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:55.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/logs
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.131:38303
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.131
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40737
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.131
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:09:55.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:55.201323 31018 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:55.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:55.201524 31018 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:55.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:55.201553 31018 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:55.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:55.203145 31018 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:55.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:55.203231 31018 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.131
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.205039 31018 tablet_server_runner.cc:78] Tablet server non-default flags:
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.131:38303
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.131
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40737
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.31018
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.131
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/logs
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:55.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:55.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.205289 31018 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:55.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.205539 31018 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:55.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:55.208355 31024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:55.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:55.208369 31026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:55.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:55.208364 31023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:55.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.208801 31018 server_base.cc:1061] running on GCE node
14:09:55.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.208946 31018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:55.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.209107 31018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:55.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.210227 31018 hybrid_clock.cc:648] HybridClock initialized: now 1783519795210201 us; error 40 us; skew 500 ppm
14:09:55.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.211244 31018 webserver.cc:533] Webserver started at http://127.29.0.131:40737/ using document root <none> and password file <none>
14:09:55.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.211411 31018 fs_manager.cc:362] Metadata directory not provided
14:09:55.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.211452 31018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:55.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.212615 31018 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:09:55.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.213246 31032 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:55.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.213461 31018 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
14:09:55.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.213531 31018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:55.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1351807c09ec4b70b1adf098a1d80986"
14:09:55.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.213822 31018 fs_report.cc:389] FS layout report
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/data
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:55.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:09:55.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.233501 31018 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:09:55.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.233884 31018 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:09:55.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.234030 31018 kserver.cc:163] Server-wide thread pool size limit: 3276
14:09:55.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.234292 31018 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:09:55.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.234779 31039 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:09:55.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.235587 31018 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:09:55.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.235646 31018 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:09:55.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.235697 31018 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:09:55.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.236320 31018 ts_tablet_manager.cc:616] Registered 1 tablets
14:09:55.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.236375 31018 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:09:55.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.236407 31039 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap starting.
14:09:55.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.243865 31018 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.131:38303
14:09:55.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.244264 31018 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:09:55.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.244432 31146 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.131:38303 every 8 connection(s)
14:09:55.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.245698 31039 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Log is configured to *not* fsync() on all Append() calls
14:09:55.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.248874 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 31018
14:09:55.249 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:09:55.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.250103 31039 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:09:55.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.250439 31149 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:09:55.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.250522 31149 heartbeater.cc:461] Registering TS with master...
14:09:55.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.250550 31039 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap complete.
14:09:55.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.250689 31149 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:09:55.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.251214 29813 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:55.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.251879 31039 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Time spent bootstrapping tablet: real 0.016s	user 0.000s	sys 0.013s
14:09:55.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.251917 31148 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:09:55.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.251977 31148 heartbeater.cc:461] Registering TS with master...
14:09:55.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.252089 31148 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:09:55.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.252380 29741 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:55.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.252741 29813 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.131:44345
14:09:55.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.253141 31039 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:55.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.253777 31039 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1351807c09ec4b70b1adf098a1d80986, State: Initialized, Role: FOLLOWER
14:09:55.253 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:09:55.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.253989 31147 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:09:55.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.254077 31147 heartbeater.cc:461] Registering TS with master...
14:09:55.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.254002 31039 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:55.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.254141 31147 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:09:55.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.254314 31039 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
14:09:55.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.254492 29886 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:55.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.254678 31149 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:09:55.255 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.0.129:37851
14:09:55.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:09:55.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:55.255780 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 30669
14:09:55.262 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:09:55.262 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] handling channelInactive
14:09:55.262 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state READY due to: connection closed
14:09:55.263 [DEBUG - Time-limited test] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:55.264 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.264 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.264 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.281 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:55.282 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:55.282 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1965847554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@799f4061 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@50fa1e9b -> wakeup thread Time-limited test)) after lookup
14:09:55.283 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:55.284 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.284 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.284 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.302 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:55.303 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:55.303 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [20ms] sending RPC to server 2d90e981114c450186889150658715c8, [22ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [22ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1965847554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@799f4061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@696159353 after retry RPC@695222539), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@50fa1e9b -> wakeup thread Time-limited test -> (continuation of Deferred@696159353 after retry RPC@695222539))) after lookup
14:09:55.304 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:55.305 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.305 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.305 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.322 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:55.323 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:55.323 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=61), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [20ms] sending RPC to server 2d90e981114c450186889150658715c8, [22ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [22ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [41ms] sending RPC to server 2d90e981114c450186889150658715c8, [43ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [43ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [60ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1965847554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@799f4061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@696159353 after retry RPC@695222539) -> (continuation of Deferred@898943811 after retry RPC@508017891), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@50fa1e9b -> wakeup thread Time-limited test -> (continuation of Deferred@696159353 after retry RPC@695222539) -> (continuation of Deferred@898943811 after retry RPC@508017891))) after lookup
14:09:55.324 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:55.325 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.325 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.325 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.361 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:55.362 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:55.362 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=100), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [20ms] sending RPC to server 2d90e981114c450186889150658715c8, [22ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [22ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [41ms] sending RPC to server 2d90e981114c450186889150658715c8, [43ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [43ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [60ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [61ms] sending RPC to server 2d90e981114c450186889150658715c8, [63ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [63ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [97ms] refreshing cache from master, [97ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [99ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1965847554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@799f4061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@696159353 after retry RPC@695222539) -> (continuation of Deferred@898943811 after retry RPC@508017891) -> (continuation of Deferred@1613967956 after retry RPC@1135618581), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@50fa1e9b -> wakeup thread Time-limited test -> (continuation of Deferred@696159353 after retry RPC@695222539) -> (continuation of Deferred@898943811 after retry RPC@508017891) -> (continuation of Deferred@1613967956 after retry RPC@1135618581))) after lookup
14:09:55.363 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:55.364 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.364 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.364 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.502 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:55.502 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:09:55.502 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=240), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [20ms] sending RPC to server 2d90e981114c450186889150658715c8, [22ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [22ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [41ms] sending RPC to server 2d90e981114c450186889150658715c8, [43ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [43ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [60ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [61ms] sending RPC to server 2d90e981114c450186889150658715c8, [63ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [63ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [97ms] refreshing cache from master, [97ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [99ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [101ms] sending RPC to server 2d90e981114c450186889150658715c8, [102ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [102ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [238ms] refreshing cache from master, [238ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [240ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1965847554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@799f4061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@696159353 after retry RPC@695222539) -> (continuation of Deferred@898943811 after retry RPC@508017891) -> (continuation of Deferred@1613967956 after retry RPC@1135618581) -> (continuation of Deferred@1785104388 after retry RPC@683993055), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@50fa1e9b -> wakeup thread Time-limited test -> (continuation of Deferred@696159353 after retry RPC@695222539) -> (continuation of Deferred@898943811 after retry RPC@508017891) -> (continuation of Deferred@1613967956 after retry RPC@1135618581) -> (continuation of Deferred@1785104388 after retry RPC@683993055))) after lookup
14:09:55.503 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:09:55.504 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.504 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:55.504 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:09:56.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.622530 31161 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2d90e981114c450186889150658715c8)
14:09:56.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.622709 31161 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:56.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.623049 31161 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:56.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:56.623466 30860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111)
14:09:56.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:56.624037 30860 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2d90e981114c450186889150658715c8 (127.29.0.129:37851): Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111)
14:09:56.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.626312 31101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "0f958fcb6937451184fba50df6455e4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "1351807c09ec4b70b1adf098a1d80986" is_pre_election: true
14:09:56.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.626464 31101 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f958fcb6937451184fba50df6455e4d in term 5.
14:09:56.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.626735 30862 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 1351807c09ec4b70b1adf098a1d80986; no voters: 2d90e981114c450186889150658715c8
14:09:56.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.626888 31161 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 6 FOLLOWER]: Leader pre-election won for term 7
14:09:56.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.626935 31161 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2d90e981114c450186889150658715c8)
14:09:56.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.626951 31161 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 6 FOLLOWER]: Advancing to term 7
14:09:56.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.627501 31161 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:56.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.627635 31161 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 7 election: Requested vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:56.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.627851 31101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "0f958fcb6937451184fba50df6455e4d" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "1351807c09ec4b70b1adf098a1d80986"
14:09:56.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.627938 31101 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 5 FOLLOWER]: Advancing to term 7
14:09:56.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:56.628273 30860 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 2d90e981114c450186889150658715c8 (127.29.0.129:37851): Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111)
14:09:56.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.628993 31101 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f958fcb6937451184fba50df6455e4d in term 7.
14:09:56.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.629173 30862 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 1351807c09ec4b70b1adf098a1d80986; no voters: 2d90e981114c450186889150658715c8
14:09:56.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.629280 31161 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 7 FOLLOWER]: Leader election won for term 7
14:09:56.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.629400 31161 raft_consensus.cc:697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 7 LEADER]: Becoming Leader. State: Replica: 0f958fcb6937451184fba50df6455e4d, State: Running, Role: LEADER
14:09:56.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.629482 31161 consensus_queue.cc:237] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:56.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:56.630124 29813 catalog_manager.cc:5697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d reported cstate change: term changed from 6 to 7, leader changed from 2d90e981114c450186889150658715c8 (127.29.0.129) to 0f958fcb6937451184fba50df6455e4d (127.29.0.130). New cstate: current_term: 7 leader_uuid: "0f958fcb6937451184fba50df6455e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } health_report { overall_health: UNKNOWN } } }
14:09:57.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.023782 31101 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 7 FOLLOWER]: Refusing update from remote peer 0f958fcb6937451184fba50df6455e4d: Log matching property violated. Preceding OpId in replica: term: 5 index: 16. Preceding OpId from leader: term: 7 index: 21. (index mismatch)
14:09:57.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.024204 31161 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
14:09:57.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:57.029243 30860 consensus_peers.cc:597] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d -> Peer 2d90e981114c450186889150658715c8 (127.29.0.129:37851): Couldn't send request to peer 2d90e981114c450186889150658715c8. Status: Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:09:57.363 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:57.363 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:57.363 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2101), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [20ms] sending RPC to server 2d90e981114c450186889150658715c8, [22ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [22ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [41ms] sending RPC to server 2d90e981114c450186889150658715c8, [43ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [43ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [60ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [61ms] sending RPC to server 2d90e981114c450186889150658715c8, [63ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [63ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [97ms] refreshing cache from master, [97ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [99ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [101ms] sending RPC to server 2d90e981114c450186889150658715c8, [102ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [102ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [238ms] refreshing cache from master, [238ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [240ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [241ms] sending RPC to server 2d90e981114c450186889150658715c8, [242ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [242ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2098ms] refreshing cache from master, [2098ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [2100ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1965847554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@799f4061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@696159353 after retry RPC@695222539) -> (continuation of Deferred@898943811 after retry RPC@508017891) -> (continuation of Deferred@1613967956 after retry RPC@1135618581) -> (continuation of Deferred@1785104388 after retry RPC@683993055) -> (continuation of Deferred@2022703618 after retry RPC@2003171536), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@50fa1e9b -> wakeup thread Time-limited test -> (continuation of Deferred@696159353 after retry RPC@695222539) -> (continuation of Deferred@898943811 after retry RPC@508017891) -> (continuation of Deferred@1613967956 after retry RPC@1135618581) -> (continuation of Deferred@1785104388 after retry RPC@683993055) -> (continuation of Deferred@2022703618 after retry RPC@2003171536))) after lookup
14:09:57.364 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:57.365 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] Successfully connected to peer
14:09:57.381 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x7971c64b, L:/127.0.0.1:48878 - R:/127.29.0.130:34547] using TOKEN/null
14:09:57.393 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.0.129:37851
14:09:57.394 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.395202 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/logs
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:57.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.129:37851
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.129
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44997
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.129
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:09:57.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:57.473052 31188 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:57.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:57.473214 31188 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:57.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:57.473235 31188 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:57.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:57.474712 31188 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:57.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:57.474762 31188 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.129
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.476367 31188 tablet_server_runner.cc:78] Tablet server non-default flags:
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.129:37851
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.129
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44997
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.31188
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.129
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/logs
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.476544 31188 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.476722 31188 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:57.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:57.479238 31195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:57.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:57.479286 31197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:57.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.479286 31188 server_base.cc:1061] running on GCE node
14:09:57.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:57.479251 31194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:57.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.479592 31188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:57.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.479851 31188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:57.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.481001 31188 hybrid_clock.cc:648] HybridClock initialized: now 1783519797480987 us; error 25 us; skew 500 ppm
14:09:57.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.482123 31188 webserver.cc:533] Webserver started at http://127.29.0.129:44997/ using document root <none> and password file <none>
14:09:57.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.482333 31188 fs_manager.cc:362] Metadata directory not provided
14:09:57.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.482402 31188 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:57.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.483566 31188 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:09:57.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.484243 31203 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:57.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.484411 31188 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
14:09:57.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.484477 31188 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:57.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2d90e981114c450186889150658715c8"
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.484810 31188 fs_report.cc:389] FS layout report
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/data
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:57.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:09:57.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.500741 31188 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:09:57.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.501107 31188 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:09:57.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.501240 31188 kserver.cc:163] Server-wide thread pool size limit: 3276
14:09:57.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.501492 31188 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:09:57.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.501996 31210 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:09:57.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.502810 31188 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:09:57.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.502887 31188 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:09:57.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.502925 31188 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:09:57.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.503517 31188 ts_tablet_manager.cc:616] Registered 1 tablets
14:09:57.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.503580 31188 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:09:57.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.503603 31210 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Bootstrap starting.
14:09:57.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.510586 31188 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.129:37851
14:09:57.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.510629 31317 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.129:37851 every 8 connection(s)
14:09:57.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.510949 31188 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:09:57.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.514160 31210 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Log is configured to *not* fsync() on all Append() calls
14:09:57.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.517854 31210 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:09:57.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.518131 31210 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Bootstrap complete.
14:09:57.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.518941 31210 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.008s
14:09:57.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.519642 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 31188
14:09:57.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.519868 31210 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:57.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.520140 31210 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d90e981114c450186889150658715c8, State: Initialized, Role: FOLLOWER
14:09:57.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.520259 31210 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:57.520 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:09:57.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.520503 31210 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
14:09:57.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.523190 31320 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:09:57.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.523300 31320 heartbeater.cc:461] Registering TS with master...
14:09:57.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.523536 31320 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:09:57.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.523793 31319 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:09:57.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.523862 31319 heartbeater.cc:461] Registering TS with master...
14:09:57.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.523963 31319 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:09:57.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.524025 31318 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:09:57.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.524075 31318 heartbeater.cc:461] Registering TS with master...
14:09:57.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.524170 31318 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:09:57.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.524262 29813 ts_manager.cc:194] Re-registered known tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:09:57.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.524317 29741 ts_manager.cc:194] Re-registered known tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:09:57.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.524753 29813 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.129:40965
14:09:57.525 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:09:57.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.525463 31320 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:09:57.526 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.0.130:34547
14:09:57.526 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:09:57.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.525907 29886 ts_manager.cc:194] Re-registered known tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:09:57.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:57.526655 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 30844
14:09:57.532 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] handling channelInactive
14:09:57.532 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state READY due to: connection closed
14:09:57.533 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:09:57.534 [DEBUG - Time-limited test] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:57.535 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.535 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.535 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.542 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:57.542 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:57.542 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [9ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1273301522(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@310a01d8 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@778b2aef -> wakeup thread Time-limited test)) after lookup
14:09:57.543 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:57.544 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.544 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.545 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.562 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:57.562 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:57.563 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [9ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [10ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [12ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [12ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [29ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1273301522(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@310a01d8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@778b2aef -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812))) after lookup
14:09:57.563 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:57.564 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.565 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.565 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.581 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:57.582 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:57.582 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=48), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [9ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [10ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [12ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [12ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [29ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [30ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [32ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [32ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [48ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1273301522(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@310a01d8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812) -> (continuation of Deferred@613735579 after retry RPC@1945015669), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@778b2aef -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812) -> (continuation of Deferred@613735579 after retry RPC@1945015669))) after lookup
14:09:57.584 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:57.585 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.585 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.585 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.681 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:57.682 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:57.682 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=149), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [9ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [10ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [12ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [12ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [29ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [30ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [32ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [32ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [48ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [50ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [52ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [52ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [148ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1273301522(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@310a01d8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812) -> (continuation of Deferred@613735579 after retry RPC@1945015669) -> (continuation of Deferred@1952256009 after retry RPC@1931437770), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@778b2aef -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812) -> (continuation of Deferred@613735579 after retry RPC@1945015669) -> (continuation of Deferred@1952256009 after retry RPC@1931437770))) after lookup
14:09:57.683 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:57.684 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.684 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:57.684 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:58.062 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:58.062 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:58.063 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=529), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [9ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [10ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [12ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [12ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [29ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [30ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [32ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [32ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [48ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [50ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [52ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [52ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [148ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [150ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [151ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [152ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [527ms] refreshing cache from master, [527ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [529ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1273301522(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@310a01d8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812) -> (continuation of Deferred@613735579 after retry RPC@1945015669) -> (continuation of Deferred@1952256009 after retry RPC@1931437770) -> (continuation of Deferred@1948006271 after retry RPC@2092774912), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@778b2aef -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812) -> (continuation of Deferred@613735579 after retry RPC@1945015669) -> (continuation of Deferred@1952256009 after retry RPC@1931437770) -> (continuation of Deferred@1948006271 after retry RPC@2092774912))) after lookup
14:09:58.063 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:58.064 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:58.064 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:58.064 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:58.802 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:58.802 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:58.802 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1269), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [9ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [10ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [12ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [12ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [29ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [30ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [32ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [32ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [48ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [50ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [52ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [52ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [148ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [150ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [151ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [152ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [527ms] refreshing cache from master, [527ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [529ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [530ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [531ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [531ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [1266ms] refreshing cache from master, [1266ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [1268ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1273301522(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@310a01d8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812) -> (continuation of Deferred@613735579 after retry RPC@1945015669) -> (continuation of Deferred@1952256009 after retry RPC@1931437770) -> (continuation of Deferred@1948006271 after retry RPC@2092774912) -> (continuation of Deferred@1626245668 after retry RPC@1970711823), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@778b2aef -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812) -> (continuation of Deferred@613735579 after retry RPC@1945015669) -> (continuation of Deferred@1952256009 after retry RPC@1931437770) -> (continuation of Deferred@1948006271 after retry RPC@2092774912) -> (continuation of Deferred@1626245668 after retry RPC@1970711823))) after lookup
14:09:58.804 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:09:58.805 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:58.805 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:58.805 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:09:58.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.870817 31332 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:09:58.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.870963 31332 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:58.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.871294 31332 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:09:58.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:58.871811 31204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:09:58.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:58.872253 31204 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547): Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:09:58.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.874497 31101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "1351807c09ec4b70b1adf098a1d80986" is_pre_election: true
14:09:58.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.874783 31207 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d90e981114c450186889150658715c8; no voters: 0f958fcb6937451184fba50df6455e4d, 1351807c09ec4b70b1adf098a1d80986
14:09:58.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.874905 31332 raft_consensus.cc:2749] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
14:09:58.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.892958 31337 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0f958fcb6937451184fba50df6455e4d)
14:09:58.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.893056 31337 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:58.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.893374 31337 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:58.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:58.893844 31033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:09:58.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:58.894331 31033 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547): Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:09:58.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.896553 31272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "1351807c09ec4b70b1adf098a1d80986" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "2d90e981114c450186889150658715c8" is_pre_election: true
14:09:58.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.896701 31272 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1351807c09ec4b70b1adf098a1d80986 in term 6.
14:09:58.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.896939 31034 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1351807c09ec4b70b1adf098a1d80986, 2d90e981114c450186889150658715c8; no voters: 0f958fcb6937451184fba50df6455e4d
14:09:58.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.897099 31337 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 7 FOLLOWER]: Leader pre-election won for term 8
14:09:58.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.897150 31337 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0f958fcb6937451184fba50df6455e4d)
14:09:58.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.897166 31337 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 7 FOLLOWER]: Advancing to term 8
14:09:58.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.897722 31337 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:58.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.897862 31337 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 8 election: Requested vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:09:58.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.898065 31272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "1351807c09ec4b70b1adf098a1d80986" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "2d90e981114c450186889150658715c8"
14:09:58.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.898164 31272 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 6 FOLLOWER]: Advancing to term 8
14:09:58.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:58.898376 31033 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547): Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:09:58.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.899418 31272 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1351807c09ec4b70b1adf098a1d80986 in term 8.
14:09:58.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.899740 31034 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1351807c09ec4b70b1adf098a1d80986, 2d90e981114c450186889150658715c8; no voters: 0f958fcb6937451184fba50df6455e4d
14:09:58.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.899860 31337 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 8 FOLLOWER]: Leader election won for term 8
14:09:58.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.899981 31337 raft_consensus.cc:697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 8 LEADER]: Becoming Leader. State: Replica: 1351807c09ec4b70b1adf098a1d80986, State: Running, Role: LEADER
14:09:58.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.900084 31337 consensus_queue.cc:237] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:09:58.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:58.900733 29813 catalog_manager.cc:5697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 reported cstate change: term changed from 7 to 8, leader changed from 0f958fcb6937451184fba50df6455e4d (127.29.0.130) to 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131). New cstate: current_term: 8 leader_uuid: "1351807c09ec4b70b1adf098a1d80986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } health_report { overall_health: HEALTHY } } }
14:09:59.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:59.417009 31033 consensus_peers.cc:597] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 -> Peer 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547): Couldn't send request to peer 0f958fcb6937451184fba50df6455e4d. Status: Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:09:59.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.484148 31272 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 8 FOLLOWER]: Refusing update from remote peer 1351807c09ec4b70b1adf098a1d80986: Log matching property violated. Preceding OpId in replica: term: 6 index: 20. Preceding OpId from leader: term: 8 index: 25. (index mismatch)
14:09:59.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.484462 31354 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
14:09:59.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.485972 31358 mvcc.cc:204] Tried to move back new op lower bound from 7305297090116767744 to 7305297086994874368. Current Snapshot: MvccSnapshot[applied={T|T < 7305297090116767744}]
14:09:59.882 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:09:59.882 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:09:59.882 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2349), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [9ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [10ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [12ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [12ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [29ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [30ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [32ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [32ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [48ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [50ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [52ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [52ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [148ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [150ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [151ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [152ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [527ms] refreshing cache from master, [527ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [529ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [530ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [531ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [531ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [1266ms] refreshing cache from master, [1266ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [1268ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [1270ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [1272ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [1272ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2346ms] refreshing cache from master, [2346ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [2348ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1273301522(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@310a01d8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812) -> (continuation of Deferred@613735579 after retry RPC@1945015669) -> (continuation of Deferred@1952256009 after retry RPC@1931437770) -> (continuation of Deferred@1948006271 after retry RPC@2092774912) -> (continuation of Deferred@1626245668 after retry RPC@1970711823) -> (continuation of Deferred@1830661394 after retry RPC@767781739), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@778b2aef -> wakeup thread Time-limited test -> (continuation of Deferred@895162522 after retry RPC@455708812) -> (continuation of Deferred@613735579 after retry RPC@1945015669) -> (continuation of Deferred@1952256009 after retry RPC@1931437770) -> (continuation of Deferred@1948006271 after retry RPC@2092774912) -> (continuation of Deferred@1626245668 after retry RPC@1970711823) -> (continuation of Deferred@1830661394 after retry RPC@767781739))) after lookup
14:09:59.884 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:09:59.885 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] Successfully connected to peer
14:09:59.897 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0xb37ef09b, L:/127.0.0.1:46996 - R:/127.29.0.131:38303] using TOKEN/null
14:09:59.907 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.0.130:34547
14:09:59.908 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.908388 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/logs
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:09:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.130:34547
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.130
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33025
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.130
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:59.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:09:59.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:59.985387 31368 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:09:59.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:59.985555 31368 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:09:59.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:59.985575 31368 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:09:59.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:59.987002 31368 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:09:59.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:59.987051 31368 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.130
14:09:59.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.988680 31368 tablet_server_runner.cc:78] Tablet server non-default flags:
14:09:59.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:09:59.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.130:34547
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.130
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33025
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.31368
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.130
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/logs
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:09:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:09:59.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:09:59.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:09:59.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.988920 31368 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:09:59.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.989107 31368 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:09:59.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:59.991825 31373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:59.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:59.991861 31376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:59.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:09:59.991825 31374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:09:59.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.992255 31368 server_base.cc:1061] running on GCE node
14:09:59.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.992396 31368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:09:59.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.992578 31368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:09:59.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.993698 31368 hybrid_clock.cc:648] HybridClock initialized: now 1783519799993672 us; error 40 us; skew 500 ppm
14:09:59.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.994822 31368 webserver.cc:533] Webserver started at http://127.29.0.130:33025/ using document root <none> and password file <none>
14:09:59.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.995046 31368 fs_manager.cc:362] Metadata directory not provided
14:09:59.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.995134 31368 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:09:59.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.996438 31368 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.000s	sys 0.000s
14:09:59.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.997192 31382 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:09:59.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.997366 31368 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
14:09:59.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.997437 31368 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:59.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0f958fcb6937451184fba50df6455e4d"
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:09:59.997816 31368 fs_report.cc:389] FS layout report
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/data
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:09:59.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:10:00.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.019950 31368 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:10:00.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.020282 31368 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:10:00.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.020412 31368 kserver.cc:163] Server-wide thread pool size limit: 3276
14:10:00.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.020668 31368 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:10:00.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.021179 31389 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:10:00.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.022105 31368 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:10:00.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.022152 31368 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:10:00.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.022178 31368 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:10:00.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.022768 31368 ts_tablet_manager.cc:616] Registered 1 tablets
14:10:00.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.022812 31368 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:10:00.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.022917 31389 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Bootstrap starting.
14:10:00.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.028402 31368 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.130:34547
14:10:00.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.028563 31496 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.130:34547 every 8 connection(s)
14:10:00.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.028841 31368 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:10:00.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.032945 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 31368
14:10:00.033 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:10:00.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.034622 31498 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:10:00.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.034693 31389 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Log is configured to *not* fsync() on all Append() calls
14:10:00.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.034720 31498 heartbeater.cc:461] Registering TS with master...
14:10:00.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.034934 31498 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:10:00.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.035521 29741 ts_manager.cc:194] Re-registered known tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:10:00.038 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:10:00.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.038803 31497 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:10:00.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.038875 31497 heartbeater.cc:461] Registering TS with master...
14:10:00.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.038975 31497 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:10:00.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.039314 29886 ts_manager.cc:194] Re-registered known tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:10:00.039 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.0.131:38303
14:10:00.040 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:10:00.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.039999 31389 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:10:00.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.040221 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 31018
14:10:00.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.040900 31389 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Bootstrap complete.
14:10:00.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.041209 31499 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:10:00.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.041301 31499 heartbeater.cc:461] Registering TS with master...
14:10:00.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.041401 31499 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:10:00.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.041716 29814 ts_manager.cc:194] Re-registered known tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:10:00.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.042009 31389 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Time spent bootstrapping tablet: real 0.019s	user 0.009s	sys 0.005s
14:10:00.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.042241 29814 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.130:52687
14:10:00.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.043560 31389 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:00.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.043905 31389 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f958fcb6937451184fba50df6455e4d, State: Initialized, Role: FOLLOWER
14:10:00.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.044018 31389 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:00.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.044273 31389 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
14:10:00.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:00.044600 31499 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:10:00.046 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:10:00.047 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] handling channelInactive
14:10:00.047 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state READY due to: connection closed
14:10:00.047 [DEBUG - Time-limited test] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:00.048 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.048 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.048 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.062 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:00.062 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:00.063 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=16), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [1ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1232874252(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@3a2a80e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@63204c34 -> wakeup thread Time-limited test)) after lookup
14:10:00.063 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:00.065 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.065 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.065 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.082 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:00.082 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:00.082 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [1ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [18ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [18ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [32ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1232874252(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@3a2a80e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@63204c34 -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394))) after lookup
14:10:00.083 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:00.084 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.084 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.084 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.121 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:00.122 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:00.122 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=75), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [1ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [18ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [18ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [32ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [35ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [37ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [37ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [74ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1232874252(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@3a2a80e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394) -> (continuation of Deferred@1372668156 after retry RPC@816221991), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@63204c34 -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394) -> (continuation of Deferred@1372668156 after retry RPC@816221991))) after lookup
14:10:00.122 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:00.123 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.123 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.124 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.181 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:00.182 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:00.182 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=135), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [1ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [18ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [18ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [32ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [35ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [37ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [37ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [74ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [75ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [77ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [77ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [134ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1232874252(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@3a2a80e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394) -> (continuation of Deferred@1372668156 after retry RPC@816221991) -> (continuation of Deferred@671462467 after retry RPC@744413651), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@63204c34 -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394) -> (continuation of Deferred@1372668156 after retry RPC@816221991) -> (continuation of Deferred@671462467 after retry RPC@744413651))) after lookup
14:10:00.183 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:00.184 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.184 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.184 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.222 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:00.223 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:00.223 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=176), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [1ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [18ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [18ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [32ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [35ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [37ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [37ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [74ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [75ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [77ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [77ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [134ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [135ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [137ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [137ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [173ms] refreshing cache from master, [173ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [175ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1232874252(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@3a2a80e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394) -> (continuation of Deferred@1372668156 after retry RPC@816221991) -> (continuation of Deferred@671462467 after retry RPC@744413651) -> (continuation of Deferred@369136028 after retry RPC@1288323790), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@63204c34 -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394) -> (continuation of Deferred@1372668156 after retry RPC@816221991) -> (continuation of Deferred@671462467 after retry RPC@744413651) -> (continuation of Deferred@369136028 after retry RPC@1288323790))) after lookup
14:10:00.224 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:00.225 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.225 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.225 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.683 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:00.683 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:00.683 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=636), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [1ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [18ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [18ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [32ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [35ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [37ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [37ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [74ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [75ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [77ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [77ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [134ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [135ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [137ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [137ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [173ms] refreshing cache from master, [173ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [175ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [177ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [178ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [179ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [633ms] refreshing cache from master, [633ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [635ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1232874252(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@3a2a80e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394) -> (continuation of Deferred@1372668156 after retry RPC@816221991) -> (continuation of Deferred@671462467 after retry RPC@744413651) -> (continuation of Deferred@369136028 after retry RPC@1288323790) -> (continuation of Deferred@433409008 after retry RPC@118994486), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@63204c34 -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394) -> (continuation of Deferred@1372668156 after retry RPC@816221991) -> (continuation of Deferred@671462467 after retry RPC@744413651) -> (continuation of Deferred@369136028 after retry RPC@1288323790) -> (continuation of Deferred@433409008 after retry RPC@118994486))) after lookup
14:10:00.684 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:00.685 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.685 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:00.685 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:01.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.325340 31510 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:10:01.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.325460 31510 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:01.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.325762 31510 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:01.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:01.326234 31386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:10:01.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:01.326716 31386 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:10:01.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.328958 31272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "0f958fcb6937451184fba50df6455e4d" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "2d90e981114c450186889150658715c8" is_pre_election: true
14:10:01.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.329267 31384 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d; no voters: 1351807c09ec4b70b1adf098a1d80986, 2d90e981114c450186889150658715c8
14:10:01.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.329411 31510 raft_consensus.cc:2749] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
14:10:01.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.407450 31515 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1351807c09ec4b70b1adf098a1d80986)
14:10:01.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.407596 31515 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:01.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.407802 31515 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:01.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:01.408879 31207 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:10:01.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.411320 31451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d" is_pre_election: true
14:10:01.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.411432 31451 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d90e981114c450186889150658715c8 in term 7.
14:10:01.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.411726 31204 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 2d90e981114c450186889150658715c8; no voters: 1351807c09ec4b70b1adf098a1d80986
14:10:01.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.411836 31515 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
14:10:01.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.411898 31515 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1351807c09ec4b70b1adf098a1d80986)
14:10:01.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.411923 31515 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 8 FOLLOWER]: Advancing to term 9
14:10:01.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.412643 31515 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:01.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.412812 31515 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 9 election: Requested vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.413008 31451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d"
14:10:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.413111 31451 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 7 FOLLOWER]: Advancing to term 9
14:10:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:01.413451 31207 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:10:01.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.414151 31451 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d90e981114c450186889150658715c8 in term 9.
14:10:01.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.414330 31204 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 2d90e981114c450186889150658715c8; no voters: 1351807c09ec4b70b1adf098a1d80986
14:10:01.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.414425 31515 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 9 FOLLOWER]: Leader election won for term 9
14:10:01.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.414669 31515 raft_consensus.cc:697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 9 LEADER]: Becoming Leader. State: Replica: 2d90e981114c450186889150658715c8, State: Running, Role: LEADER
14:10:01.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.414772 31515 consensus_queue.cc:237] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:01.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.415426 29814 catalog_manager.cc:5697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 reported cstate change: term changed from 8 to 9, leader changed from 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131) to 2d90e981114c450186889150658715c8 (127.29.0.129). New cstate: current_term: 9 leader_uuid: "2d90e981114c450186889150658715c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } health_report { overall_health: UNKNOWN } } }
14:10:01.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:01.888795 31207 consensus_peers.cc:597] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 -> Peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Couldn't send request to peer 1351807c09ec4b70b1adf098a1d80986. Status: Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:10:01.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.916309 31451 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 9 FOLLOWER]: Refusing update from remote peer 2d90e981114c450186889150658715c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 24. Preceding OpId from leader: term: 9 index: 29. (index mismatch)
14:10:01.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:01.916684 31515 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
14:10:03.702 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:03.702 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:10:03.703 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=3655), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [1ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [1ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [16ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [18ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [18ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [32ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [35ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [37ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [37ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [74ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [75ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [77ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [77ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [134ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [135ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [137ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [137ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [173ms] refreshing cache from master, [173ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [175ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [177ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [178ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [179ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [633ms] refreshing cache from master, [633ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [635ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [637ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [638ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [638ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [3653ms] refreshing cache from master, [3653ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [3655ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@1232874252(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@3a2a80e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394) -> (continuation of Deferred@1372668156 after retry RPC@816221991) -> (continuation of Deferred@671462467 after retry RPC@744413651) -> (continuation of Deferred@369136028 after retry RPC@1288323790) -> (continuation of Deferred@433409008 after retry RPC@118994486) -> (continuation of Deferred@494650074 after retry RPC@669479237), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@63204c34 -> wakeup thread Time-limited test -> (continuation of Deferred@2125426991 after retry RPC@529519394) -> (continuation of Deferred@1372668156 after retry RPC@816221991) -> (continuation of Deferred@671462467 after retry RPC@744413651) -> (continuation of Deferred@369136028 after retry RPC@1288323790) -> (continuation of Deferred@433409008 after retry RPC@118994486) -> (continuation of Deferred@494650074 after retry RPC@669479237))) after lookup
14:10:03.704 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:10:03.705 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] Successfully connected to peer
14:10:03.730 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x678994dd, L:/127.0.0.1:35856 - R:/127.29.0.129:37851] using TOKEN/null
14:10:03.740 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.0.131:38303
14:10:03.740 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:10:03.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.741235 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:10:03.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:10:03.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:10:03.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data
14:10:03.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:10:03.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:10:03.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:10:03.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:10:03.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/logs
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.131:38303
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.131
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40737
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.131
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:10:03.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:10:03.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:03.819581 31553 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:10:03.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:03.819772 31553 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:10:03.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:03.819792 31553 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:10:03.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:03.821408 31553 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:10:03.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:03.821466 31553 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.131
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.822956 31553 tablet_server_runner.cc:78] Tablet server non-default flags:
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.131:38303
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.131
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40737
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.31553
14:10:03.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.131
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/logs
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.823175 31553 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:10:03.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.823381 31553 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:10:03.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:03.826052 31558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:03.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:03.826164 31559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:03.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.826344 31553 server_base.cc:1061] running on GCE node
14:10:03.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:03.826084 31561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:03.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.826733 31553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:10:03.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.826941 31553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:10:03.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.828083 31553 hybrid_clock.cc:648] HybridClock initialized: now 1783519803828058 us; error 39 us; skew 500 ppm
14:10:03.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.829272 31553 webserver.cc:533] Webserver started at http://127.29.0.131:40737/ using document root <none> and password file <none>
14:10:03.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.829427 31553 fs_manager.cc:362] Metadata directory not provided
14:10:03.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.829463 31553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:10:03.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.830615 31553 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:10:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.831190 31567 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:10:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.831362 31553 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
14:10:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.831430 31553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:10:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1351807c09ec4b70b1adf098a1d80986"
14:10:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:10:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.831735 31553 fs_report.cc:389] FS layout report
14:10:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:10:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:10:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:10:03.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/data
14:10:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:10:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:10:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:10:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:10:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:10:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:10:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:10:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:10:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:10:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:10:03.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:10:03.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.860054 31553 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:10:03.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.860312 31553 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:10:03.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.860396 31553 kserver.cc:163] Server-wide thread pool size limit: 3276
14:10:03.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.860577 31553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:10:03.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.860981 31574 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:10:03.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.861891 31553 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:10:03.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.861938 31553 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:10:03.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.861964 31553 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:10:03.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.862509 31553 ts_tablet_manager.cc:616] Registered 1 tablets
14:10:03.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.862540 31553 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:10:03.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.862630 31574 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap starting.
14:10:03.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.869715 31553 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.131:38303
14:10:03.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.869799 31681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.131:38303 every 8 connection(s)
14:10:03.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.870124 31553 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:10:03.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.873734 31574 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Log is configured to *not* fsync() on all Append() calls
14:10:03.876 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:10:03.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.876122 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 31553
14:10:03.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.876590 31682 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:10:03.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.876675 31682 heartbeater.cc:461] Registering TS with master...
14:10:03.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.876848 31682 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:10:03.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.877061 31684 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:10:03.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.877118 31684 heartbeater.cc:461] Registering TS with master...
14:10:03.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.877215 31684 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:10:03.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.877400 29887 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:03.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.877822 29813 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:03.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.878036 31574 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:10:03.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.878262 29813 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.131:39661
14:10:03.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.878322 31574 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap complete.
14:10:03.880 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:10:03.881 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.0.129:37851
14:10:03.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.879347 31683 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:10:03.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.879402 31683 heartbeater.cc:461] Registering TS with master...
14:10:03.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.879436 31574 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Time spent bootstrapping tablet: real 0.017s	user 0.002s	sys 0.012s
14:10:03.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.879513 31683 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:10:03.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.880050 29740 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:03.881 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:10:03.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.880614 31574 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:03.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.880861 31574 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1351807c09ec4b70b1adf098a1d80986, State: Initialized, Role: FOLLOWER
14:10:03.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.880957 31574 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:03.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.881194 31574 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
14:10:03.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.881444 31684 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:10:03.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:03.881717 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 31188
14:10:03.888 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:10:03.888 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] handling channelInactive
14:10:03.888 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state READY due to: connection closed
14:10:03.889 [DEBUG - Time-limited test] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:10:03.890 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:03.890 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:03.890 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:03.902 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:03.902 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:10:03.902 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@284339665(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@6e11359b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@205067de -> wakeup thread Time-limited test)) after lookup
14:10:03.903 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:10:03.904 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:03.904 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:03.904 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:03.921 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:03.922 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:10:03.922 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [15ms] sending RPC to server 2d90e981114c450186889150658715c8, [16ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@284339665(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@6e11359b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@205067de -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760))) after lookup
14:10:03.923 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:10:03.924 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:03.924 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:03.924 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:03.941 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:03.942 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:10:03.942 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=53), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [15ms] sending RPC to server 2d90e981114c450186889150658715c8, [16ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [34ms] sending RPC to server 2d90e981114c450186889150658715c8, [36ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [36ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [53ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@284339665(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@6e11359b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@205067de -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231))) after lookup
14:10:03.942 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:10:03.943 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:03.943 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:03.944 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:04.002 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:04.002 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:10:04.002 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=113), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [15ms] sending RPC to server 2d90e981114c450186889150658715c8, [16ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [34ms] sending RPC to server 2d90e981114c450186889150658715c8, [36ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [36ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [53ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [54ms] sending RPC to server 2d90e981114c450186889150658715c8, [56ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [56ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [111ms] refreshing cache from master, [112ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@284339665(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@6e11359b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231) -> (continuation of Deferred@358673050 after retry RPC@1183194746), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@205067de -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231) -> (continuation of Deferred@358673050 after retry RPC@1183194746))) after lookup
14:10:04.003 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:10:04.004 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:04.004 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:04.004 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:04.362 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:04.362 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:10:04.362 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=473), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [15ms] sending RPC to server 2d90e981114c450186889150658715c8, [16ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [34ms] sending RPC to server 2d90e981114c450186889150658715c8, [36ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [36ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [53ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [54ms] sending RPC to server 2d90e981114c450186889150658715c8, [56ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [56ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [111ms] refreshing cache from master, [112ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [115ms] sending RPC to server 2d90e981114c450186889150658715c8, [116ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [116ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [472ms] refreshing cache from master, [472ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [473ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@284339665(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@6e11359b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231) -> (continuation of Deferred@358673050 after retry RPC@1183194746) -> (continuation of Deferred@234797296 after retry RPC@2071790331), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@205067de -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231) -> (continuation of Deferred@358673050 after retry RPC@1183194746) -> (continuation of Deferred@234797296 after retry RPC@2071790331))) after lookup
14:10:04.363 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:10:04.364 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:04.364 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:04.364 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:04.822 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:04.822 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:10:04.823 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=934), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [15ms] sending RPC to server 2d90e981114c450186889150658715c8, [16ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [34ms] sending RPC to server 2d90e981114c450186889150658715c8, [36ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [36ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [53ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [54ms] sending RPC to server 2d90e981114c450186889150658715c8, [56ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [56ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [111ms] refreshing cache from master, [112ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [115ms] sending RPC to server 2d90e981114c450186889150658715c8, [116ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [116ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [472ms] refreshing cache from master, [472ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [473ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [475ms] sending RPC to server 2d90e981114c450186889150658715c8, [476ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [476ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [931ms] refreshing cache from master, [932ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [934ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@284339665(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@6e11359b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231) -> (continuation of Deferred@358673050 after retry RPC@1183194746) -> (continuation of Deferred@234797296 after retry RPC@2071790331) -> (continuation of Deferred@1119168379 after retry RPC@629779402), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@205067de -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231) -> (continuation of Deferred@358673050 after retry RPC@1183194746) -> (continuation of Deferred@234797296 after retry RPC@2071790331) -> (continuation of Deferred@1119168379 after retry RPC@629779402))) after lookup
14:10:04.823 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:10:04.824 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:04.824 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:04.824 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:05.121 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:05.121 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:10:05.122 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=14, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1233), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [15ms] sending RPC to server 2d90e981114c450186889150658715c8, [16ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [34ms] sending RPC to server 2d90e981114c450186889150658715c8, [36ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [36ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [53ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [54ms] sending RPC to server 2d90e981114c450186889150658715c8, [56ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [56ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [111ms] refreshing cache from master, [112ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [115ms] sending RPC to server 2d90e981114c450186889150658715c8, [116ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [116ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [472ms] refreshing cache from master, [472ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [473ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [475ms] sending RPC to server 2d90e981114c450186889150658715c8, [476ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [476ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [931ms] refreshing cache from master, [932ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [934ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [935ms] sending RPC to server 2d90e981114c450186889150658715c8, [936ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [936ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [1231ms] refreshing cache from master, [1231ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [1233ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@284339665(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@6e11359b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231) -> (continuation of Deferred@358673050 after retry RPC@1183194746) -> (continuation of Deferred@234797296 after retry RPC@2071790331) -> (continuation of Deferred@1119168379 after retry RPC@629779402) -> (continuation of Deferred@1048212216 after retry RPC@1447385350), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@205067de -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231) -> (continuation of Deferred@358673050 after retry RPC@1183194746) -> (continuation of Deferred@234797296 after retry RPC@2071790331) -> (continuation of Deferred@1119168379 after retry RPC@629779402) -> (continuation of Deferred@1048212216 after retry RPC@1447385350))) after lookup
14:10:05.122 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:10:05.123 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:05.124 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:05.124 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 2d90e981114c450186889150658715c8(127.29.0.129:37851) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851
14:10:05.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.240566 31696 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 2d90e981114c450186889150658715c8)
14:10:05.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.240679 31696 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:05.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.240854 31696 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:05.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:05.241668 31384 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 2d90e981114c450186889150658715c8 (127.29.0.129:37851): Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111)
14:10:05.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.244231 31636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "0f958fcb6937451184fba50df6455e4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "1351807c09ec4b70b1adf098a1d80986" is_pre_election: true
14:10:05.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.244359 31636 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f958fcb6937451184fba50df6455e4d in term 8.
14:10:05.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.244607 31386 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 1351807c09ec4b70b1adf098a1d80986; no voters: 2d90e981114c450186889150658715c8
14:10:05.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.244717 31696 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 9 FOLLOWER]: Leader pre-election won for term 10
14:10:05.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.244779 31696 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2d90e981114c450186889150658715c8)
14:10:05.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.244807 31696 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 9 FOLLOWER]: Advancing to term 10
14:10:05.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.245599 31696 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:05.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.245779 31696 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 10 election: Requested vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:05.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.245966 31636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "0f958fcb6937451184fba50df6455e4d" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "1351807c09ec4b70b1adf098a1d80986"
14:10:05.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.246052 31636 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 8 FOLLOWER]: Advancing to term 10
14:10:05.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:05.246541 31384 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 2d90e981114c450186889150658715c8 (127.29.0.129:37851): Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111)
14:10:05.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.247398 31636 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f958fcb6937451184fba50df6455e4d in term 10.
14:10:05.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.247602 31386 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 1351807c09ec4b70b1adf098a1d80986; no voters: 2d90e981114c450186889150658715c8
14:10:05.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.247695 31696 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 10 FOLLOWER]: Leader election won for term 10
14:10:05.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.247885 31696 raft_consensus.cc:697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 10 LEADER]: Becoming Leader. State: Replica: 0f958fcb6937451184fba50df6455e4d, State: Running, Role: LEADER
14:10:05.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.247997 31696 consensus_queue.cc:237] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:05.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.248724 29813 catalog_manager.cc:5697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d reported cstate change: term changed from 9 to 10, leader changed from 2d90e981114c450186889150658715c8 (127.29.0.129) to 0f958fcb6937451184fba50df6455e4d (127.29.0.130). New cstate: current_term: 10 leader_uuid: "0f958fcb6937451184fba50df6455e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } health_report { overall_health: UNKNOWN } } }
14:10:05.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.667519 31636 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 10 FOLLOWER]: Refusing update from remote peer 0f958fcb6937451184fba50df6455e4d: Log matching property violated. Preceding OpId in replica: term: 8 index: 28. Preceding OpId from leader: term: 10 index: 33. (index mismatch)
14:10:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:05.667860 31696 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
14:10:05.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:05.670635 31384 consensus_peers.cc:597] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d -> Peer 2d90e981114c450186889150658715c8 (127.29.0.129:37851): Couldn't send request to peer 2d90e981114c450186889150658715c8. Status: Network error: Client connection negotiation failed: client connection to 127.29.0.129:37851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:10:06.022 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:06.022 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:06.023 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=16, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2134), Traces: [0ms] sending RPC to server 2d90e981114c450186889150658715c8, [2ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [15ms] sending RPC to server 2d90e981114c450186889150658715c8, [16ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [34ms] sending RPC to server 2d90e981114c450186889150658715c8, [36ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [36ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [53ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [54ms] sending RPC to server 2d90e981114c450186889150658715c8, [56ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [56ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [111ms] refreshing cache from master, [112ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [115ms] sending RPC to server 2d90e981114c450186889150658715c8, [116ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [116ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [472ms] refreshing cache from master, [472ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [473ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [475ms] sending RPC to server 2d90e981114c450186889150658715c8, [476ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [476ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [931ms] refreshing cache from master, [932ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [934ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [935ms] sending RPC to server 2d90e981114c450186889150658715c8, [936ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [936ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [1231ms] refreshing cache from master, [1231ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [1233ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [1234ms] sending RPC to server 2d90e981114c450186889150658715c8, [1236ms] received response from server 2d90e981114c450186889150658715c8: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [1236ms] delaying RPC due to: Network error: Failed to connect to peer 2d90e981114c450186889150658715c8(127.29.0.129:37851): Connection refused: /127.29.0.129:37851, [2132ms] refreshing cache from master, [2132ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [2134ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@284339665(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@6e11359b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231) -> (continuation of Deferred@358673050 after retry RPC@1183194746) -> (continuation of Deferred@234797296 after retry RPC@2071790331) -> (continuation of Deferred@1119168379 after retry RPC@629779402) -> (continuation of Deferred@1048212216 after retry RPC@1447385350) -> (continuation of Deferred@520387290 after retry RPC@1668652373), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@205067de -> wakeup thread Time-limited test -> (continuation of Deferred@420940251 after retry RPC@987427760) -> (continuation of Deferred@1405588900 after retry RPC@1055763231) -> (continuation of Deferred@358673050 after retry RPC@1183194746) -> (continuation of Deferred@234797296 after retry RPC@2071790331) -> (continuation of Deferred@1119168379 after retry RPC@629779402) -> (continuation of Deferred@1048212216 after retry RPC@1447385350) -> (continuation of Deferred@520387290 after retry RPC@1668652373))) after lookup
14:10:06.024 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:10:06.024 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] Successfully connected to peer
14:10:06.038 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xcf395b74, L:/127.0.0.1:56694 - R:/127.29.0.130:34547] using TOKEN/null
14:10:06.047 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.0.129:37851
14:10:06.047 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.048184 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/logs
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:10:06.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.129:37851
14:10:06.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.129
14:10:06.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:10:06.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44997
14:10:06.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.129
14:10:06.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:10:06.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:10:06.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:10:06.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:10:06.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:06.124779 31723 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:10:06.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:06.124948 31723 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:10:06.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:06.124966 31723 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:10:06.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:06.126382 31723 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:10:06.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:06.126437 31723 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.129
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.127928 31723 tablet_server_runner.cc:78] Tablet server non-default flags:
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.129:37851
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.129
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44997
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.31723
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.129
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/logs
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.128101 31723 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:10:06.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.128265 31723 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:10:06.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:06.130755 31729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:06.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.130838 31723 server_base.cc:1061] running on GCE node
14:10:06.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:06.130831 31731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:06.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:06.131503 31728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:06.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.131773 31723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:10:06.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.132104 31723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:10:06.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.133356 31723 hybrid_clock.cc:648] HybridClock initialized: now 1783519806133314 us; error 62 us; skew 500 ppm
14:10:06.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.134650 31723 webserver.cc:533] Webserver started at http://127.29.0.129:44997/ using document root <none> and password file <none>
14:10:06.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.134910 31723 fs_manager.cc:362] Metadata directory not provided
14:10:06.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.134990 31723 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:10:06.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.136338 31723 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.000s	sys 0.000s
14:10:06.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.137116 31737 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:10:06.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.137337 31723 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
14:10:06.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.137416 31723 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:10:06.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2d90e981114c450186889150658715c8"
14:10:06.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.137820 31723 fs_report.cc:389] FS layout report
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/wal
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/data
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:10:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:10:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.151456 31723 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:10:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.151814 31723 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:10:06.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.151973 31723 kserver.cc:163] Server-wide thread pool size limit: 3276
14:10:06.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.152175 31723 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:10:06.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.152617 31744 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:10:06.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.153410 31723 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:10:06.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.153529 31723 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:10:06.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.153569 31723 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:10:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.154132 31723 ts_tablet_manager.cc:616] Registered 1 tablets
14:10:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.154191 31723 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:10:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.154234 31744 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Bootstrap starting.
14:10:06.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.161090 31723 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.129:37851
14:10:06.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.161130 31851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.129:37851 every 8 connection(s)
14:10:06.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.161437 31723 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-0/data/info.pb
14:10:06.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.162370 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 31723
14:10:06.163 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:10:06.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.167128 31744 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Log is configured to *not* fsync() on all Append() calls
14:10:06.169 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:10:06.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.168603 31853 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:10:06.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.168691 31853 heartbeater.cc:461] Registering TS with master...
14:10:06.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.168859 31853 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:10:06.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.169404 29741 ts_manager.cc:194] Re-registered known tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:10:06.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.171965 31852 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:10:06.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.172020 31852 heartbeater.cc:461] Registering TS with master...
14:10:06.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.172123 31852 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:10:06.172 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.0.130:34547
14:10:06.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.172361 31744 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:10:06.172 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:10:06.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.172673 31744 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Bootstrap complete.
14:10:06.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.172871 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 31368
14:10:06.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.172998 29887 ts_manager.cc:194] Re-registered known tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:10:06.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.174197 31744 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Time spent bootstrapping tablet: real 0.020s	user 0.007s	sys 0.009s
14:10:06.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.175707 31744 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:06.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.175987 31744 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d90e981114c450186889150658715c8, State: Initialized, Role: FOLLOWER
14:10:06.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.176081 31744 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:06.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.176301 31744 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
14:10:06.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.177937 31854 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:10:06.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.177996 31854 heartbeater.cc:461] Registering TS with master...
14:10:06.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.178088 31854 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:10:06.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.178373 29813 ts_manager.cc:194] Re-registered known tserver with Master: 2d90e981114c450186889150658715c8 (127.29.0.129:37851)
14:10:06.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.178705 29813 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.129:43521
14:10:06.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:06.179177 31854 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:10:06.181 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:10:06.181 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] handling channelInactive
14:10:06.181 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state READY due to: connection closed
14:10:06.181 [DEBUG - Time-limited test] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:10:06.182 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.183 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.183 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.202 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:06.203 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:06.203 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=22), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [21ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@822861775(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@494f621a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7678bd78 -> wakeup thread Time-limited test)) after lookup
14:10:06.204 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:10:06.205 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.205 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.205 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.221 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:06.221 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:06.222 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [21ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [22ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [24ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [24ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@822861775(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@494f621a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1811134890 after retry RPC@1481853649), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7678bd78 -> wakeup thread Time-limited test -> (continuation of Deferred@1811134890 after retry RPC@1481853649))) after lookup
14:10:06.222 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:10:06.223 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.223 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.224 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.262 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:06.262 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:06.262 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=81), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [21ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [22ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [24ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [24ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [41ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [43ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [43ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@822861775(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@494f621a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1811134890 after retry RPC@1481853649) -> (continuation of Deferred@320419917 after retry RPC@1365179709), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7678bd78 -> wakeup thread Time-limited test -> (continuation of Deferred@1811134890 after retry RPC@1481853649) -> (continuation of Deferred@320419917 after retry RPC@1365179709))) after lookup
14:10:06.263 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:10:06.264 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.264 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.264 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.401 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:06.401 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:06.402 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=220), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [21ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [22ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [24ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [24ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [41ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [43ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [43ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [82ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [83ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [83ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [218ms] refreshing cache from master, [218ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [220ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@822861775(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@494f621a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1811134890 after retry RPC@1481853649) -> (continuation of Deferred@320419917 after retry RPC@1365179709) -> (continuation of Deferred@1100014971 after retry RPC@1736531809), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7678bd78 -> wakeup thread Time-limited test -> (continuation of Deferred@1811134890 after retry RPC@1481853649) -> (continuation of Deferred@320419917 after retry RPC@1365179709) -> (continuation of Deferred@1100014971 after retry RPC@1736531809))) after lookup
14:10:06.402 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:10:06.403 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.403 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.403 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.542 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:06.542 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)[L],1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:06.542 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=361), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [21ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [22ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [24ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [24ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [41ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [43ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [43ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [82ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [83ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [83ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [218ms] refreshing cache from master, [218ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [220ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [221ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [222ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [222ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [358ms] refreshing cache from master, [359ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [360ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@822861775(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@494f621a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1811134890 after retry RPC@1481853649) -> (continuation of Deferred@320419917 after retry RPC@1365179709) -> (continuation of Deferred@1100014971 after retry RPC@1736531809) -> (continuation of Deferred@2140041730 after retry RPC@312033477), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7678bd78 -> wakeup thread Time-limited test -> (continuation of Deferred@1811134890 after retry RPC@1481853649) -> (continuation of Deferred@320419917 after retry RPC@1365179709) -> (continuation of Deferred@1100014971 after retry RPC@1736531809) -> (continuation of Deferred@2140041730 after retry RPC@312033477))) after lookup
14:10:06.543 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] connecting to peer
14:10:06.544 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.544 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] cleaning up while in state CONNECTING due to: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:06.544 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547
14:10:07.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.547070 31866 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0f958fcb6937451184fba50df6455e4d)
14:10:07.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.547183 31866 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:07.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.547504 31866 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:10:07.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.547986 31568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:10:07.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.548482 31568 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547): Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:10:07.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.550645 31806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "1351807c09ec4b70b1adf098a1d80986" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "2d90e981114c450186889150658715c8" is_pre_election: true
14:10:07.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.550817 31806 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1351807c09ec4b70b1adf098a1d80986 in term 9.
14:10:07.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.551060 31569 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1351807c09ec4b70b1adf098a1d80986, 2d90e981114c450186889150658715c8; no voters: 0f958fcb6937451184fba50df6455e4d
14:10:07.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.551262 31866 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 10 FOLLOWER]: Leader pre-election won for term 11
14:10:07.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.551319 31866 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0f958fcb6937451184fba50df6455e4d)
14:10:07.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.551335 31866 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 10 FOLLOWER]: Advancing to term 11
14:10:07.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.552201 31866 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:07.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.552351 31866 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 11 election: Requested vote from peers 2d90e981114c450186889150658715c8 (127.29.0.129:37851), 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:10:07.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.552621 31806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "1351807c09ec4b70b1adf098a1d80986" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "2d90e981114c450186889150658715c8"
14:10:07.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.552716 31806 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 9 FOLLOWER]: Advancing to term 11
14:10:07.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.553138 31568 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547): Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111)
14:10:07.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.553996 31806 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1351807c09ec4b70b1adf098a1d80986 in term 11.
14:10:07.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.554201 31569 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1351807c09ec4b70b1adf098a1d80986, 2d90e981114c450186889150658715c8; no voters: 0f958fcb6937451184fba50df6455e4d
14:10:07.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.554314 31866 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 11 FOLLOWER]: Leader election won for term 11
14:10:07.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.554471 31866 raft_consensus.cc:697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 11 LEADER]: Becoming Leader. State: Replica: 1351807c09ec4b70b1adf098a1d80986, State: Running, Role: LEADER
14:10:07.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.554567 31866 consensus_queue.cc:237] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:07.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.555272 29813 catalog_manager.cc:5697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 reported cstate change: term changed from 10 to 11, leader changed from 0f958fcb6937451184fba50df6455e4d (127.29.0.130) to 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131). New cstate: current_term: 11 leader_uuid: "1351807c09ec4b70b1adf098a1d80986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } health_report { overall_health: HEALTHY } } }
14:10:07.882 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:07.882 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:07.882 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1701), Traces: [0ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [2ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [2ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [21ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [22ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [24ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [24ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [41ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [43ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [43ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [82ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [83ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [83ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [218ms] refreshing cache from master, [218ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [220ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [221ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [222ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [222ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [358ms] refreshing cache from master, [359ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [360ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [362ms] sending RPC to server 0f958fcb6937451184fba50df6455e4d, [363ms] received response from server 0f958fcb6937451184fba50df6455e4d: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [363ms] delaying RPC due to: Network error: Failed to connect to peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547): Connection refused: /127.29.0.130:34547, [1698ms] refreshing cache from master, [1698ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [1700ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@822861775(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@494f621a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1811134890 after retry RPC@1481853649) -> (continuation of Deferred@320419917 after retry RPC@1365179709) -> (continuation of Deferred@1100014971 after retry RPC@1736531809) -> (continuation of Deferred@2140041730 after retry RPC@312033477) -> (continuation of Deferred@1194601764 after retry RPC@536068719), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7678bd78 -> wakeup thread Time-limited test -> (continuation of Deferred@1811134890 after retry RPC@1481853649) -> (continuation of Deferred@320419917 after retry RPC@1365179709) -> (continuation of Deferred@1100014971 after retry RPC@1736531809) -> (continuation of Deferred@2140041730 after retry RPC@312033477) -> (continuation of Deferred@1194601764 after retry RPC@536068719))) after lookup
14:10:07.883 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:07.885 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] Successfully connected to peer
14:10:07.900 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0xd50fe6ea, L:/127.0.0.1:49652 - R:/127.29.0.131:38303] using TOKEN/null
14:10:07.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.901803 31806 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 11 FOLLOWER]: Refusing update from remote peer 1351807c09ec4b70b1adf098a1d80986: Log matching property violated. Preceding OpId in replica: term: 9 index: 32. Preceding OpId from leader: term: 11 index: 38. (index mismatch)
14:10:07.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.902079 31568 consensus_peers.cc:597] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 -> Peer 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547): Couldn't send request to peer 0f958fcb6937451184fba50df6455e4d. Status: Network error: Client connection negotiation failed: client connection to 127.29.0.130:34547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:10:07.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.902117 31866 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
14:10:07.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.904009 31884 mvcc.cc:204] Tried to move back new op lower bound from 7305297133163159552 to 7305297131744083968. Current Snapshot: MvccSnapshot[applied={T|T < 7305297133163159552}]
14:10:07.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.904968 31886 mvcc.cc:204] Tried to move back new op lower bound from 7305297133163159552 to 7305297131744083968. Current Snapshot: MvccSnapshot[applied={T|T < 7305297133163159552}]
14:10:07.914 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.0.130:34547
14:10:07.914 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.915174 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/logs
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:10:07.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.130:34547
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.130
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33025
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.130
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:10:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.992204 31893 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.992391 31893 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.992422 31893 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:10:07.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.993997 31893 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:10:07.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.994084 31893 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.130
14:10:07.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.995637 31893 tablet_server_runner.cc:78] Tablet server non-default flags:
14:10:07.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:10:07.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:10:07.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.130:34547
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.130
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33025
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.31893
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.130
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/logs
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.995894 31893 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:10:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.996134 31893 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:10:07.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.998890 31899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:07.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.998915 31898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:07.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:07.998915 31901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:07.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.999634 31893 server_base.cc:1061] running on GCE node
14:10:07.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:07.999867 31893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:10:08.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.000094 31893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:10:08.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.001240 31893 hybrid_clock.cc:648] HybridClock initialized: now 1783519808001228 us; error 26 us; skew 500 ppm
14:10:08.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.002381 31893 webserver.cc:533] Webserver started at http://127.29.0.130:33025/ using document root <none> and password file <none>
14:10:08.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.002612 31893 fs_manager.cc:362] Metadata directory not provided
14:10:08.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.002688 31893 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:10:08.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.003863 31893 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.000s	sys 0.001s
14:10:08.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.004498 31907 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:10:08.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.004658 31893 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.001s
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.004719 31893 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0f958fcb6937451184fba50df6455e4d"
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.004954 31893 fs_report.cc:389] FS layout report
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/wal
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/data
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:10:08.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.015197 31893 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:10:08.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.015470 31893 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:10:08.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.015555 31893 kserver.cc:163] Server-wide thread pool size limit: 3276
14:10:08.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.015789 31893 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:10:08.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.016289 31914 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:10:08.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.017071 31893 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:10:08.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.017130 31893 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:10:08.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.017164 31893 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:10:08.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.017715 31893 ts_tablet_manager.cc:616] Registered 1 tablets
14:10:08.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.017771 31893 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:10:08.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.017793 31914 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Bootstrap starting.
14:10:08.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.025543 31893 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.130:34547
14:10:08.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.025626 32021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.130:34547 every 8 connection(s)
14:10:08.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.025996 31893 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-1/data/info.pb
14:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.027240 31914 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Log is configured to *not* fsync() on all Append() calls
14:10:08.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.029708 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 31893
14:10:08.031 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:10:08.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.031944 31914 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:10:08.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.032259 31914 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Bootstrap complete.
14:10:08.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.032306 32023 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:10:08.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.032374 32023 heartbeater.cc:461] Registering TS with master...
14:10:08.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.032558 32023 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:10:08.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.033098 29741 ts_manager.cc:194] Re-registered known tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:10:08.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.033416 31914 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.008s
14:10:08.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.034606 31914 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:08.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.034875 31914 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f958fcb6937451184fba50df6455e4d, State: Initialized, Role: FOLLOWER
14:10:08.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.034981 31914 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:08.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.035202 31914 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
14:10:08.037 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:10:08.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.037425 32024 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:10:08.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.037497 32024 heartbeater.cc:461] Registering TS with master...
14:10:08.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.037611 32024 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:10:08.038 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.0.131:38303
14:10:08.038 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:10:08.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.037918 29814 ts_manager.cc:194] Re-registered known tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:10:08.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.038445 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 31553
14:10:08.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.038460 29814 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.130:47329
14:10:08.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.040001 32022 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:10:08.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.040074 32022 heartbeater.cc:461] Registering TS with master...
14:10:08.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.040120 32024 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:10:08.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.040127 32022 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:10:08.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:08.040577 29887 ts_manager.cc:194] Re-registered known tserver with Master: 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547)
14:10:08.044 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:10:08.045 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] handling channelInactive
14:10:08.045 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state READY due to: connection closed
14:10:08.060 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:08.062 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.062 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.062 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.081 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:08.082 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:08.082 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=3, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=37), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [17ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [17ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [36ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@492735798(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@245f22c2 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2197a68d -> wakeup thread Time-limited test)) after lookup
14:10:08.083 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:08.084 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.084 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.084 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.102 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:08.102 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:08.103 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=5, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=57), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [17ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [17ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [36ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [37ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [39ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [39ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [57ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@492735798(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@245f22c2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@461391813 after retry RPC@121580022), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2197a68d -> wakeup thread Time-limited test -> (continuation of Deferred@461391813 after retry RPC@121580022))) after lookup
14:10:08.103 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:08.104 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.104 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.104 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.121 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:08.122 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:08.122 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=7, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=77), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [17ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [17ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [36ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [37ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [39ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [39ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [57ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [58ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [59ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [59ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [74ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [76ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@492735798(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@245f22c2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@461391813 after retry RPC@121580022) -> (continuation of Deferred@119589436 after retry RPC@2041455543), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2197a68d -> wakeup thread Time-limited test -> (continuation of Deferred@461391813 after retry RPC@121580022) -> (continuation of Deferred@119589436 after retry RPC@2041455543))) after lookup
14:10:08.123 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:08.124 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.124 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.124 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.281 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:08.281 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:08.282 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=9, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=236), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [17ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [17ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [36ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [37ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [39ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [39ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [57ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [58ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [59ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [59ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [74ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [76ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [78ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [79ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [79ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [234ms] refreshing cache from master, [234ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [236ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@492735798(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@245f22c2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@461391813 after retry RPC@121580022) -> (continuation of Deferred@119589436 after retry RPC@2041455543) -> (continuation of Deferred@977935649 after retry RPC@846279236), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2197a68d -> wakeup thread Time-limited test -> (continuation of Deferred@461391813 after retry RPC@121580022) -> (continuation of Deferred@119589436 after retry RPC@2041455543) -> (continuation of Deferred@977935649 after retry RPC@846279236))) after lookup
14:10:08.282 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:08.284 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.284 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.284 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.742 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:08.742 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)[L],2d90e981114c450186889150658715c8(127.29.0.129:37851)]}]
14:10:08.742 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=11, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=697), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [17ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [17ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [36ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [37ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [39ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [39ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [57ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [58ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [59ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [59ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [74ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [76ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [78ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [79ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [79ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [234ms] refreshing cache from master, [234ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [236ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [237ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [239ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [239ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [694ms] refreshing cache from master, [695ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [697ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@492735798(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@245f22c2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@461391813 after retry RPC@121580022) -> (continuation of Deferred@119589436 after retry RPC@2041455543) -> (continuation of Deferred@977935649 after retry RPC@846279236) -> (continuation of Deferred@1703992454 after retry RPC@1581789538), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2197a68d -> wakeup thread Time-limited test -> (continuation of Deferred@461391813 after retry RPC@121580022) -> (continuation of Deferred@119589436 after retry RPC@2041455543) -> (continuation of Deferred@977935649 after retry RPC@846279236) -> (continuation of Deferred@1703992454 after retry RPC@1581789538))) after lookup
14:10:08.744 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] connecting to peer
14:10:08.745 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.745 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:08.745 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303) for tablet 5b5fca86382a4532a7aaef4dc5ca1391: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303
14:10:09.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.414778 32035 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1351807c09ec4b70b1adf098a1d80986)
14:10:09.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.414883 32035 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:09.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.415166 32035 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:09.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:09.415704 31741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:10:09.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:09.416179 31741 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:10:09.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.418596 31976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d" is_pre_election: true
14:10:09.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.418721 31976 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d90e981114c450186889150658715c8 in term 10.
14:10:09.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.418965 31738 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 2d90e981114c450186889150658715c8; no voters: 1351807c09ec4b70b1adf098a1d80986
14:10:09.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.419126 32035 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
14:10:09.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.419198 32035 raft_consensus.cc:493] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1351807c09ec4b70b1adf098a1d80986)
14:10:09.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.419220 32035 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 11 FOLLOWER]: Advancing to term 12
14:10:09.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.420078 32035 raft_consensus.cc:515] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:09.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.420228 32035 leader_election.cc:290] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 12 election: Requested vote from peers 0f958fcb6937451184fba50df6455e4d (127.29.0.130:34547), 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:09.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.420387 31976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b5fca86382a4532a7aaef4dc5ca1391" candidate_uuid: "2d90e981114c450186889150658715c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "0f958fcb6937451184fba50df6455e4d"
14:10:09.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.420548 31976 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 10 FOLLOWER]: Advancing to term 12
14:10:09.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:09.421015 31741 leader_election.cc:336] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111)
14:10:09.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.421726 31976 raft_consensus.cc:2468] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d90e981114c450186889150658715c8 in term 12.
14:10:09.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.421948 31738 leader_election.cc:304] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f958fcb6937451184fba50df6455e4d, 2d90e981114c450186889150658715c8; no voters: 1351807c09ec4b70b1adf098a1d80986
14:10:09.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.422057 32035 raft_consensus.cc:2804] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 12 FOLLOWER]: Leader election won for term 12
14:10:09.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.422232 32035 raft_consensus.cc:697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [term 12 LEADER]: Becoming Leader. State: Replica: 2d90e981114c450186889150658715c8, State: Running, Role: LEADER
14:10:09.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.422322 32035 consensus_queue.cc:237] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:09.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:09.423144 29814 catalog_manager.cc:5697] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 reported cstate change: term changed from 11 to 12, leader changed from 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131) to 2d90e981114c450186889150658715c8 (127.29.0.129). New cstate: current_term: 12 leader_uuid: "2d90e981114c450186889150658715c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } health_report { overall_health: UNKNOWN } } }
14:10:09.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:09.937060 31741 consensus_peers.cc:597] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 -> Peer 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303): Couldn't send request to peer 1351807c09ec4b70b1adf098a1d80986. Status: Network error: Client connection negotiation failed: client connection to 127.29.0.131:38303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:10:10.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:10.031567 31976 raft_consensus.cc:1275] T 5b5fca86382a4532a7aaef4dc5ca1391 P 0f958fcb6937451184fba50df6455e4d [term 12 FOLLOWER]: Refusing update from remote peer 2d90e981114c450186889150658715c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 36. Preceding OpId from leader: term: 12 index: 41. (index mismatch)
14:10:10.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:10.032109 32052 consensus_queue.cc:1048] T 5b5fca86382a4532a7aaef4dc5ca1391 P 2d90e981114c450186889150658715c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
14:10:11.162 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 5b5fca86382a4532a7aaef4dc5ca1391 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106' with partition [<start>, <end>)
14:10:11.162 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=5b5fca86382a4532a7aaef4dc5ca1391@[0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547),1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303),2d90e981114c450186889150658715c8(127.29.0.129:37851)[L]]}]
14:10:11.162 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=5b5fca86382a4532a7aaef4dc5ca1391, attempt=13, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=3117), Traces: [0ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [17ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [17ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [36ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [37ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [39ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [39ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [57ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [58ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [59ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [59ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [74ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [76ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [78ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [79ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [79ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [234ms] refreshing cache from master, [234ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [236ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [237ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [239ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [239ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [694ms] refreshing cache from master, [695ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [697ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, [699ms] sending RPC to server 1351807c09ec4b70b1adf098a1d80986, [700ms] received response from server 1351807c09ec4b70b1adf098a1d80986: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [700ms] delaying RPC due to: Network error: Failed to connect to peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303): Connection refused: /127.29.0.131:38303, [3114ms] refreshing cache from master, [3114ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.0.189:46475, [3116ms] Sub RPC GetTableLocations: received response from server master-127.29.0.189:46475: OK, deferred=Deferred@492735798(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007fb10c339420@245f22c2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@461391813 after retry RPC@121580022) -> (continuation of Deferred@119589436 after retry RPC@2041455543) -> (continuation of Deferred@977935649 after retry RPC@846279236) -> (continuation of Deferred@1703992454 after retry RPC@1581789538) -> (continuation of Deferred@457866477 after retry RPC@68586951), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2197a68d -> wakeup thread Time-limited test -> (continuation of Deferred@461391813 after retry RPC@121580022) -> (continuation of Deferred@119589436 after retry RPC@2041455543) -> (continuation of Deferred@977935649 after retry RPC@846279236) -> (continuation of Deferred@1703992454 after retry RPC@1581789538) -> (continuation of Deferred@457866477 after retry RPC@68586951))) after lookup
14:10:11.163 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] connecting to peer
14:10:11.164 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] Successfully connected to peer
14:10:11.191 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x94dbcb10, L:/127.0.0.1:35932 - R:/127.29.0.129:37851] using TOKEN/null
14:10:11.208 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.0.131:38303
14:10:11.208 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.209193 29698 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGaLwZW/build/release/bin/kudu
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/logs
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:10:11.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.131:38303
14:10:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.131
14:10:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:10:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40737
14:10:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.131
14:10:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:10:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:10:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:10:11.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:10:11.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:11.286057 32074 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:10:11.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:11.286254 32074 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:10:11.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:11.286285 32074 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:10:11.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:11.287992 32074 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:10:11.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:11.288077 32074 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.0.131
14:10:11.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.289637 32074 tablet_server_runner.cc:78] Tablet server non-default flags:
14:10:11.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:10:11.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.0.148:36137
14:10:11.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:10:11.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.0.131:38303
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.0.131
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40737
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.0.190:45183,127.29.0.189:46475,127.29.0.188:40013
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.32074
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.0.131
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/logs
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:15 UTC on e7f111948823
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12773
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.289899 32074 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:10:11.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.290151 32074 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:10:11.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:11.292963 32079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:11.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:11.293072 32082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:11.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.293201 32074 server_base.cc:1061] running on GCE node
14:10:11.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:10:11.292943 32080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:10:11.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.293578 32074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:10:11.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.293805 32074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:10:11.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.294967 32074 hybrid_clock.cc:648] HybridClock initialized: now 1783519811294948 us; error 34 us; skew 500 ppm
14:10:11.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.296195 32074 webserver.cc:533] Webserver started at http://127.29.0.131:40737/ using document root <none> and password file <none>
14:10:11.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.296346 32074 fs_manager.cc:362] Metadata directory not provided
14:10:11.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.296380 32074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:10:11.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.297502 32074 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.298094 32088 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.298242 32074 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.298292 32074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data,/tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1351807c09ec4b70b1adf098a1d80986"
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:09:39 on dist-test-slave-0nsb"
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.298555 32074 fs_report.cc:389] FS layout report
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/wal
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/data
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:10:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:10:11.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:10:11.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:10:11.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:10:11.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:10:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.307435 32074 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:10:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.307744 32074 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:10:11.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.307865 32074 kserver.cc:163] Server-wide thread pool size limit: 3276
14:10:11.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.308096 32074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:10:11.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.308568 32095 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:10:11.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.309509 32074 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:10:11.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.309569 32074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:10:11.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.309603 32074 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:10:11.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.310153 32074 ts_tablet_manager.cc:616] Registered 1 tablets
14:10:11.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.310207 32074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:10:11.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.310256 32095 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap starting.
14:10:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.316802 32074 rpc_server.cc:307] RPC server started. Bound to: 127.29.0.131:38303
14:10:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.316847 32202 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.0.131:38303 every 8 connection(s)
14:10:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.317154 32074 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskGaLwZW/test-tmp/mini-kudu-cluster12896885987241125302/ts-2/data/info.pb
14:10:11.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.321669 32095 log.cc:826] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Log is configured to *not* fsync() on all Append() calls
14:10:11.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.322742 32203 heartbeater.cc:344] Connected to a master server at 127.29.0.188:40013
14:10:11.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.322748 32204 heartbeater.cc:344] Connected to a master server at 127.29.0.190:45183
14:10:11.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.322885 32204 heartbeater.cc:461] Registering TS with master...
14:10:11.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.322885 32203 heartbeater.cc:461] Registering TS with master...
14:10:11.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.323083 32203 heartbeater.cc:507] Master 127.29.0.188:40013 requested a full tablet report, sending...
14:10:11.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.323082 32204 heartbeater.cc:507] Master 127.29.0.190:45183 requested a full tablet report, sending...
14:10:11.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.323578 29698 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskGaLwZW/build/release/bin/kudu as pid 32074
14:10:11.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.323655 29741 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:11.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.323899 29887 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:11.324 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:10:11.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.327175 32095 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:10:11.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.327476 32095 tablet_bootstrap.cc:492] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Bootstrap complete.
14:10:11.327 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:10:11.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.328549 32095 ts_tablet_manager.cc:1403] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.003s
14:10:11.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.329725 32095 raft_consensus.cc:359] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:11.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.329993 32095 raft_consensus.cc:740] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1351807c09ec4b70b1adf098a1d80986, State: Initialized, Role: FOLLOWER
14:10:11.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.330262 32095 consensus_queue.cc:260] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d90e981114c450186889150658715c8" member_type: VOTER last_known_addr { host: "127.29.0.129" port: 37851 } } peers { permanent_uuid: "0f958fcb6937451184fba50df6455e4d" member_type: VOTER last_known_addr { host: "127.29.0.130" port: 34547 } } peers { permanent_uuid: "1351807c09ec4b70b1adf098a1d80986" member_type: VOTER last_known_addr { host: "127.29.0.131" port: 38303 } }
14:10:11.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.330524 32095 ts_tablet_manager.cc:1434] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
14:10:11.332 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519780106, tablet=5b5fca86382a4532a7aaef4dc5ca1391, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:10:11.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.336495 32205 heartbeater.cc:344] Connected to a master server at 127.29.0.189:46475
14:10:11.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.336570 32205 heartbeater.cc:461] Registering TS with master...
14:10:11.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.336670 32205 heartbeater.cc:507] Master 127.29.0.189:46475 requested a full tablet report, sending...
14:10:11.337 [DEBUG - main] (Connection.java:575) [peer master-127.29.0.190:45183(127.29.0.190:45183)] disconnecting while in state READY
14:10:11.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.336937 29813 ts_manager.cc:194] Re-registered known tserver with Master: 1351807c09ec4b70b1adf098a1d80986 (127.29.0.131:38303)
14:10:11.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.337486 29813 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.0.131:51605
14:10:11.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.338297 32205 heartbeater.cc:499] Master 127.29.0.189:46475 was elected leader, sending a full tablet report...
14:10:11.340 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.29.0.190:45183(127.29.0.190:45183)] handling channelInactive
14:10:11.341 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.29.0.190:45183(127.29.0.190:45183)] cleaning up while in state READY due to: connection closed
14:10:11.342 [DEBUG - main] (Connection.java:575) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] disconnecting while in state READY
14:10:11.342 [DEBUG - main] (Connection.java:575) [peer master-127.29.0.188:40013(127.29.0.188:40013)] disconnecting while in state READY
14:10:11.343 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] handling channelInactive
14:10:11.343 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 2d90e981114c450186889150658715c8(127.29.0.129:37851)] cleaning up while in state READY due to: connection closed
14:10:11.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.340615 32157 raft_consensus.cc:3060] T 5b5fca86382a4532a7aaef4dc5ca1391 P 1351807c09ec4b70b1adf098a1d80986 [term 11 FOLLOWER]: Advancing to term 12
14:10:11.344 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.29.0.188:40013(127.29.0.188:40013)] handling channelInactive
14:10:11.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.345085 32217 mvcc.cc:204] Tried to move back new op lower bound from 7305297146673528832 to 7305297139394428928. Current Snapshot: MvccSnapshot[applied={T|T < 7305297146645331968}]
14:10:11.345 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.29.0.188:40013(127.29.0.188:40013)] cleaning up while in state READY due to: connection closed
14:10:11.345 [DEBUG - main] (Connection.java:575) [peer master-127.29.0.189:46475(127.29.0.189:46475)] disconnecting while in state READY
14:10:11.346 [DEBUG - main] (Connection.java:575) [peer 0f958fcb6937451184fba50df6455e4d(127.29.0.130:34547)] disconnecting while in state TERMINATED
14:10:11.346 [DEBUG - main] (Connection.java:575) [peer 1351807c09ec4b70b1adf098a1d80986(127.29.0.131:38303)] disconnecting while in state TERMINATED
14:10:11.346 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.29.0.189:46475(127.29.0.189:46475)] handling channelInactive
14:10:11.346 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
14:10:11.347 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.29.0.189:46475(127.29.0.189:46475)] cleaning up while in state READY due to: connection closed
14:10:11.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.352720 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 31723
14:10:11.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.358544 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 31893
14:10:11.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.363415 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 32074
14:10:11.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.369267 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 29710
14:10:11.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.374444 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 29781
14:10:11.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:10:11.379083 29698 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskGaLwZW/build/release/bin/kudu with pid 29854
14:10:11.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T14:10:11Z chronyd exiting

Time: 33.935

OK (1 test)

