JUnit version 4.13.2
.07:56:42.999 [INFO - main] (RandomUtils.java:49) Using random seed: 1783497402992
07:56:43.047 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
07:56:43.114 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
07:56:43.130 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskJhaYq9/test-tmp
07:56:43.133 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/release/bin
07:56:43.136 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskJhaYq9/java/build/dist-test/../../../build/release/bin/kudu, test, mini_cluster, --serialization=pb]
07:56:43.362 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

07:56:43.375 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
07:56:43.376 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

07:56:43.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.377235  4725 mini_oidc.cc:152] Starting JWKS server
07:56:43.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.378301  4725 webserver.cc:460] Webserver started at http://127.0.0.1:33577/ using document root <none> and password file <none>
07:56:43.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.378407  4725 mini_oidc.cc:189] Starting OIDC Discovery server
07:56:43.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.378585  4725 webserver.cc:460] Webserver started at http://127.0.0.1:46511/ using document root <none> and password file <none>
07:56:43.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T07:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
07:56:43.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T07:56:43Z Disabled control of system clock
07:56:43.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.392357  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:43.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:43.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/wal
07:56:43.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/data
07:56:43.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:43.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:43.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:43.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:43.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:43.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:43.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/logs
07:56:43.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/data/info.pb
07:56:43.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:43.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:43.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:43.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:43.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:43.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:43.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:43.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:43.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
07:56:43.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:43.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:43.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:43.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.126:38135
07:56:43.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.126
07:56:43.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:43.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:43.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:43.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:43.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:43.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:43.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:43.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:43.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33577/jwks.json with env {}
07:56:43.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.471374  4736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:43.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.471616  4736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:43.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.471648  4736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:43.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.473238  4736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
07:56:43.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.473309  4736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:43.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.473333  4736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
07:56:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.473353  4736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
07:56:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.473373  4736 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
07:56:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.473388  4736 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:33577/jwks.json
07:56:43.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.475299  4736 master_runner.cc:386] Master server non-default flags:
07:56:43.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:43.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:43.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:43.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:43.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/data
07:56:43.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/wal
07:56:43.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:43.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:43.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:43.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:43.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:43.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.126:38135
07:56:43.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:43.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:43.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:43.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33577/jwks.json
07:56:43.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:43.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:43.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/data/info.pb
07:56:43.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.126
07:56:43.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:43.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.4736
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/logs
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:43.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:43.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:43.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:43.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:43.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.475589  4736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:43.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.476043  4736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.483026  4741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.483108  4744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.483150  4736 server_base.cc:1034] running on GCE node
07:56:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.483026  4742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:43.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.483604  4736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:43.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.483960  4736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:43.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.485071  4736 hybrid_clock.cc:648] HybridClock initialized: now 1783497403485053 us; error 34 us; skew 500 ppm
07:56:43.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.486673  4736 webserver.cc:460] Webserver started at http://127.4.157.126:34573/ using document root <none> and password file <none>
07:56:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.487016  4736 fs_manager.cc:359] Metadata directory not provided
07:56:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.487080  4736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.487241  4736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
07:56:43.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.488660  4736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/data/instance:
07:56:43.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a23c643f60244d58b7af6e724e7fbb05"
07:56:43.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:43 on dist-test-slave-5j59"
07:56:43.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.489187  4736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/wal/instance:
07:56:43.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a23c643f60244d58b7af6e724e7fbb05"
07:56:43.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:43 on dist-test-slave-5j59"
07:56:43.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.490902  4736 fs_manager.cc:691] Time spent creating directory manager: real 0.002s	user 0.000s	sys 0.002s
07:56:43.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.491811  4750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:43.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.492004  4736 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
07:56:43.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.492086  4736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/wal
07:56:43.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a23c643f60244d58b7af6e724e7fbb05"
07:56:43.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:43 on dist-test-slave-5j59"
07:56:43.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.492134  4736 fs_report.cc:383] FS layout report
07:56:43.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:43.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/wal
07:56:43.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/wal
07:56:43.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/data/data
07:56:43.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:43.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:43.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:43.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:43.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:43.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:43.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:43.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:43.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:43.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:43.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.504339  4736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.504617  4736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.504711  4736 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:43.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.508366  4736 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.126:38135
07:56:43.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.508466  4802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.126:38135 every 8 connection(s)
07:56:43.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.508675  4736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/data/info.pb
07:56:43.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.509419  4803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:43.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.511543  4803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } has no permanent_uuid. Determining permanent_uuid...
07:56:43.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.517270  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 4736
07:56:43.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.517400  4725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-0/wal/instance
07:56:43.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.517359  4803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } has no permanent_uuid. Determining permanent_uuid...
07:56:43.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.517792  4753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.125:45997: connect: Connection refused (error 111)
07:56:43.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.518203  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:43.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:43.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/wal
07:56:43.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/data
07:56:43.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:43.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:43.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:43.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:43.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:43.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:43.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/logs
07:56:43.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/data/info.pb
07:56:43.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:43.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:43.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:43.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:43.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:43.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:43.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:43.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:43.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
07:56:43.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:43.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:43.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:43.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.125:45997
07:56:43.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.125
07:56:43.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:43.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:43.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:43.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:43.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:43.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:43.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:43.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:43.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33577/jwks.json with env {}
07:56:43.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.518395  4803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.157.125:45997: Network error: Client connection negotiation failed: client connection to 127.4.157.125:45997: connect: Connection refused (error 111)
07:56:43.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.567631  4803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } attempt: 1
07:56:43.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.568645  4803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.157.125:45997: Network error: Client connection negotiation failed: client connection to 127.4.157.125:45997: connect: Connection refused (error 111)
07:56:43.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.601173  4808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:43.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.601397  4808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:43.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.601418  4808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:43.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.602931  4808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
07:56:43.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.602979  4808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:43.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.602993  4808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
07:56:43.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.603006  4808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
07:56:43.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.603019  4808 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
07:56:43.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.603030  4808 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:33577/jwks.json
07:56:43.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.605032  4808 master_runner.cc:386] Master server non-default flags:
07:56:43.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:43.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:43.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:43.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:43.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/data
07:56:43.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/wal
07:56:43.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:43.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:43.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:43.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:43.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:43.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.125:45997
07:56:43.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:43.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:43.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33577/jwks.json
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/data/info.pb
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.125
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.4808
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:43.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/logs
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.605288  4808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.605631  4808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:43.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.611618  4816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.611627  4813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.611699  4808 server_base.cc:1034] running on GCE node
07:56:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.611702  4814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:43.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.612105  4808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:43.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.612462  4808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:43.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.613607  4808 hybrid_clock.cc:648] HybridClock initialized: now 1783497403613586 us; error 34 us; skew 500 ppm
07:56:43.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.614934  4808 webserver.cc:460] Webserver started at http://127.4.157.125:34215/ using document root <none> and password file <none>
07:56:43.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.615217  4808 fs_manager.cc:359] Metadata directory not provided
07:56:43.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.615342  4808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:43.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.615485  4808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
07:56:43.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.616503  4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/data/instance:
07:56:43.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "76435a19a5854cb48b74edc491ebe9f9"
07:56:43.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:43 on dist-test-slave-5j59"
07:56:43.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.616892  4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/wal/instance:
07:56:43.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "76435a19a5854cb48b74edc491ebe9f9"
07:56:43.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:43 on dist-test-slave-5j59"
07:56:43.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.618479  4808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.001s
07:56:43.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.619349  4822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:43.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.619601  4808 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
07:56:43.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.619692  4808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/wal
07:56:43.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "76435a19a5854cb48b74edc491ebe9f9"
07:56:43.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:43 on dist-test-slave-5j59"
07:56:43.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.619763  4808 fs_report.cc:383] FS layout report
07:56:43.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/wal
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/wal
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/data/data
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:43.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.632578  4808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:43.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.632953  4808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:43.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.633142  4808 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:43.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.636812  4803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } attempt: 2
07:56:43.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.637532  4808 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.125:45997
07:56:43.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.637560  4874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.125:45997 every 8 connection(s)
07:56:43.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.638131  4808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/data/info.pb
07:56:43.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.638846  4876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.640457  4876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } has no permanent_uuid. Determining permanent_uuid...
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.643850  4803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } has no permanent_uuid. Determining permanent_uuid...
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.644151  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 4808
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.644232  4725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-1/wal/instance
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.644484  4803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.157.124:46883: Network error: Client connection negotiation failed: client connection to 127.4.157.124:46883: connect: Connection refused (error 111)
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.644618  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/wal
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/data
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:43.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/logs
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/data/info.pb
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:43.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.124:46883
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.124
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:43.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:43.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33577/jwks.json with env {}
07:56:43.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.645915  4876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } has no permanent_uuid. Determining permanent_uuid...
07:56:43.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.649700  4876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } has no permanent_uuid. Determining permanent_uuid...
07:56:43.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.650099  4825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.124:46883: connect: Connection refused (error 111)
07:56:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.651042  4876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.157.124:46883: Network error: Client connection negotiation failed: client connection to 127.4.157.124:46883: connect: Connection refused (error 111)
07:56:43.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.687644  4803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } attempt: 1
07:56:43.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.689029  4803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.157.124:46883: Network error: Client connection negotiation failed: client connection to 127.4.157.124:46883: connect: Connection refused (error 111)
07:56:43.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.700276  4876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } attempt: 1
07:56:43.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.701124  4876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.157.124:46883: Network error: Client connection negotiation failed: client connection to 127.4.157.124:46883: connect: Connection refused (error 111)
07:56:43.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.731940  4879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:43.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.732136  4879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:43.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.732160  4879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:43.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.733283  4879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
07:56:43.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.733322  4879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:43.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.733335  4879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
07:56:43.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.733348  4879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
07:56:43.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.733363  4879 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
07:56:43.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.733379  4879 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:33577/jwks.json
07:56:43.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.734673  4879 master_runner.cc:386] Master server non-default flags:
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/data
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/wal
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:56:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.124:46883
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:33577/jwks.json
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/data/info.pb
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.124
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.4879
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/logs
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.734876  4879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.735168  4879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:43.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.736842  4803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } attempt: 2
07:56:43.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.737599  4803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.157.124:46883: Network error: Client connection negotiation failed: client connection to 127.4.157.124:46883: connect: Connection refused (error 111)
07:56:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.741142  4886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.741106  4885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.741107  4888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.741406  4879 server_base.cc:1034] running on GCE node
07:56:43.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.741689  4879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:43.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.742033  4879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:43.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.743204  4879 hybrid_clock.cc:648] HybridClock initialized: now 1783497403743189 us; error 39 us; skew 500 ppm
07:56:43.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.744793  4879 webserver.cc:460] Webserver started at http://127.4.157.124:34775/ using document root <none> and password file <none>
07:56:43.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.745090  4879 fs_manager.cc:359] Metadata directory not provided
07:56:43.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.745157  4879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:43.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.745302  4879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
07:56:43.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.746536  4879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/data/instance:
07:56:43.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d06b7df478984ce3a473ce69c83a5f7d"
07:56:43.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:43 on dist-test-slave-5j59"
07:56:43.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.746951  4879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/wal/instance:
07:56:43.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d06b7df478984ce3a473ce69c83a5f7d"
07:56:43.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:43 on dist-test-slave-5j59"
07:56:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.748219  4879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s	user 0.003s	sys 0.000s
07:56:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.749020  4894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.749190  4879 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
07:56:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.749308  4879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/wal
07:56:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d06b7df478984ce3a473ce69c83a5f7d"
07:56:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:43 on dist-test-slave-5j59"
07:56:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.749377  4879 fs_report.cc:383] FS layout report
07:56:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/wal
07:56:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/wal
07:56:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/data/data
07:56:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:43.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.759884  4879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:43.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.760214  4879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:43.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.760404  4879 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:43.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.764429  4879 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.124:46883
07:56:43.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.764508  4946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.124:46883 every 8 connection(s)
07:56:43.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.764853  4879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/data/info.pb
07:56:43.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.765580  4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:43.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.767443  4947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } has no permanent_uuid. Determining permanent_uuid...
07:56:43.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.769299  4876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } attempt: 2
07:56:43.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.771093  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 4879
07:56:43.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.771185  4725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/master-2/wal/instance
07:56:43.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.775586  4876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9: Bootstrap starting.
07:56:43.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.776376  4876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9: Neither blocks nor log segments found. Creating new log.
07:56:43.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.776726  4876 log.cc:824] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9: Log is configured to *not* fsync() on all Append() calls
07:56:43.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.777045  4947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } has no permanent_uuid. Determining permanent_uuid...
07:56:43.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.778267  4876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9: No bootstrap required, opened a new log
07:56:43.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.780702  4876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } }
07:56:43.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.780835  4876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:43.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.780874  4876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76435a19a5854cb48b74edc491ebe9f9, State: Initialized, Role: FOLLOWER
07:56:43.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.780990  4876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } }
07:56:43.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.781319  4876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [sys.catalog]: configured and running, proceeding with master startup.
07:56:43.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.782913  4947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } has no permanent_uuid. Determining permanent_uuid...
07:56:43.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.783245  4967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
07:56:43.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.783465  4967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
07:56:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.786319  4956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } } }
07:56:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.786721  4956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [sys.catalog]: This master's current role is: FOLLOWER
07:56:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.783815  4808 master_runner.cc:186] Error getting master registration for 127.4.157.126:38135: OK, instance_id { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" instance_seqno: 1783497403507420 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
07:56:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.787720  4808 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
07:56:43.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.790009  4947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d: Bootstrap starting.
07:56:43.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.790838  4947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d: Neither blocks nor log segments found. Creating new log.
07:56:43.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.791177  4947 log.cc:824] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d: Log is configured to *not* fsync() on all Append() calls
07:56:43.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.792008  4947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d: No bootstrap required, opened a new log
07:56:43.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.793825  4947 raft_consensus.cc:348] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } }
07:56:43.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.793960  4947 raft_consensus.cc:374] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:43.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.793998  4947 raft_consensus.cc:729] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06b7df478984ce3a473ce69c83a5f7d, State: Initialized, Role: FOLLOWER
07:56:43.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.794132  4947 consensus_queue.cc:260] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } }
07:56:43.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.794512  4947 sys_catalog.cc:564] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [sys.catalog]: configured and running, proceeding with master startup.
07:56:43.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.794459  4970 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } } }
07:56:43.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.794598  4970 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [sys.catalog]: This master's current role is: FOLLOWER
07:56:43.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.796535  4981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
07:56:43.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.796772  4981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
07:56:43.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.799343  4879 master_runner.cc:186] Error getting master registration for 127.4.157.126:38135: OK, instance_id { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" instance_seqno: 1783497403507420 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
07:56:43.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.802067  4879 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
07:56:43.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.844833  4803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } attempt: 3
07:56:43.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.851011  4803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05: Bootstrap starting.
07:56:43.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.851771  4803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05: Neither blocks nor log segments found. Creating new log.
07:56:43.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.852478  4803 log.cc:824] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05: Log is configured to *not* fsync() on all Append() calls
07:56:43.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.853637  4803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05: No bootstrap required, opened a new log
07:56:43.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.855628  4803 raft_consensus.cc:348] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } }
07:56:43.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.855755  4803 raft_consensus.cc:374] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:43.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.855787  4803 raft_consensus.cc:729] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23c643f60244d58b7af6e724e7fbb05, State: Initialized, Role: FOLLOWER
07:56:43.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.855918  4803 consensus_queue.cc:260] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } }
07:56:43.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.856235  4803 sys_catalog.cc:564] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [sys.catalog]: configured and running, proceeding with master startup.
07:56:43.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.858071  4995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
07:56:43.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.858342  4995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
07:56:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.860131  4736 master_runner.cc:263] No leader master found from master a23c643f60244d58b7af6e724e7fbb05
07:56:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.860178  4736 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
07:56:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.860730  4984 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } } }
07:56:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.861027  4984 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [sys.catalog]: This master's current role is: FOLLOWER
07:56:43.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:43.883879  4828 tablet.cc:2369] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:56:43.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.912322  4956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:56:43.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.912446  4956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } }
07:56:43.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.912679  4956 leader_election.cc:288] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23c643f60244d58b7af6e724e7fbb05 (127.4.157.126:38135), d06b7df478984ce3a473ce69c83a5f7d (127.4.157.124:46883)
07:56:43.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.913056  4921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76435a19a5854cb48b74edc491ebe9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06b7df478984ce3a473ce69c83a5f7d" is_pre_election: true
07:56:43.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.913205  4921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76435a19a5854cb48b74edc491ebe9f9 in term 0.
07:56:43.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.913376  4825 leader_election.cc:302] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76435a19a5854cb48b74edc491ebe9f9, d06b7df478984ce3a473ce69c83a5f7d; no voters: 
07:56:43.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.913728  4777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76435a19a5854cb48b74edc491ebe9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23c643f60244d58b7af6e724e7fbb05" is_pre_election: true
07:56:43.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.913872  4777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76435a19a5854cb48b74edc491ebe9f9 in term 0.
07:56:43.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.915436  4956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.915504  4956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.915529  4956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [term 0 FOLLOWER]: Advancing to term 1
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.916426  4956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } }
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.916550  4956 leader_election.cc:288] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [CANDIDATE]: Term 1 election: Requested vote from peers a23c643f60244d58b7af6e724e7fbb05 (127.4.157.126:38135), d06b7df478984ce3a473ce69c83a5f7d (127.4.157.124:46883)
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.916822  4921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76435a19a5854cb48b74edc491ebe9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06b7df478984ce3a473ce69c83a5f7d"
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.916895  4921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [term 0 FOLLOWER]: Advancing to term 1
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.917078  4777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76435a19a5854cb48b74edc491ebe9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23c643f60244d58b7af6e724e7fbb05"
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.917148  4777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [term 0 FOLLOWER]: Advancing to term 1
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.917996  4777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76435a19a5854cb48b74edc491ebe9f9 in term 1.
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.918165  4824 leader_election.cc:302] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76435a19a5854cb48b74edc491ebe9f9, a23c643f60244d58b7af6e724e7fbb05; no voters: 
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.918910  4921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76435a19a5854cb48b74edc491ebe9f9 in term 1.
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.919960  4956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [term 1 FOLLOWER]: Leader election won for term 1
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.920125  4956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 76435a19a5854cb48b74edc491ebe9f9, State: Running, Role: LEADER
07:56:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.920213  4956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } }
07:56:43.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.920574  4956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76435a19a5854cb48b74edc491ebe9f9. Latest consensus state: current_term: 1 leader_uuid: "76435a19a5854cb48b74edc491ebe9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } } }
07:56:43.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.920644  4956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [sys.catalog]: This master's current role is: LEADER
07:56:43.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.922547  4999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
07:56:43.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.922971  4999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
07:56:43.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.925767  4921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [term 1 FOLLOWER]: Refusing update from remote peer 76435a19a5854cb48b74edc491ebe9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:56:43.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.926303  4956 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:56:43.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.929925  4970 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76435a19a5854cb48b74edc491ebe9f9. Latest consensus state: current_term: 1 leader_uuid: "76435a19a5854cb48b74edc491ebe9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } } }
07:56:43.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.930042  4970 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [sys.catalog]: This master's current role is: FOLLOWER
07:56:43.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.934317  4777 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [term 1 FOLLOWER]: Refusing update from remote peer 76435a19a5854cb48b74edc491ebe9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:56:43.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.934625  4956 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:56:43.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.935951  4984 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76435a19a5854cb48b74edc491ebe9f9. Latest consensus state: current_term: 1 leader_uuid: "76435a19a5854cb48b74edc491ebe9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } } }
07:56:43.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.936048  4984 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [sys.catalog]: This master's current role is: FOLLOWER
07:56:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.940429  4997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76435a19a5854cb48b74edc491ebe9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } } }
07:56:43.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.940526  4997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [sys.catalog]: This master's current role is: LEADER
07:56:43.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.941504  4984 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76435a19a5854cb48b74edc491ebe9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } } }
07:56:43.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.941581  4984 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05 [sys.catalog]: This master's current role is: FOLLOWER
07:56:43.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.944635  5000 mvcc.cc:205] Tried to move back new op lower bound from 7305205366477418496 to 7305205366457651200. Current Snapshot: MvccSnapshot[applied={T|T < 7305205366477418496}]
07:56:43.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.945510  5003 mvcc.cc:205] Tried to move back new op lower bound from 7305205366477418496 to 7305205366457651200. Current Snapshot: MvccSnapshot[applied={T|T < 7305205366477418496}]
07:56:43.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.946601  4956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76435a19a5854cb48b74edc491ebe9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } } }
07:56:43.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.946708  4956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9 [sys.catalog]: This master's current role is: LEADER
07:56:43.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.947083  4970 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76435a19a5854cb48b74edc491ebe9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23c643f60244d58b7af6e724e7fbb05" member_type: VOTER last_known_addr { host: "127.4.157.126" port: 38135 } } peers { permanent_uuid: "76435a19a5854cb48b74edc491ebe9f9" member_type: VOTER last_known_addr { host: "127.4.157.125" port: 45997 } } peers { permanent_uuid: "d06b7df478984ce3a473ce69c83a5f7d" member_type: VOTER last_known_addr { host: "127.4.157.124" port: 46883 } } }
07:56:43.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.947170  4970 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d [sys.catalog]: This master's current role is: FOLLOWER
07:56:43.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.948324  5001 mvcc.cc:205] Tried to move back new op lower bound from 7305205366477418496 to 7305205366457651200. Current Snapshot: MvccSnapshot[applied={T|T < 7305205366477418496}]
07:56:43.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.949465  4999 catalog_manager.cc:1347] Generated new cluster ID: a42f5a84af0049a8aeb85a93fc647cad
07:56:43.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.949510  4999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
07:56:43.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.972398  4999 catalog_manager.cc:1370] Generated new certificate authority record
07:56:43.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.973268  4999 catalog_manager.cc:1504] Loading token signing keys...
07:56:43.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.976963  4999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76435a19a5854cb48b74edc491ebe9f9: Generated new TSK 0
07:56:43.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.977385  4999 catalog_manager.cc:1514] Initializing in-progress tserver states...
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:43.995872  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/logs
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.65:0
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.65
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.65
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:44.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.082433  5009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:44.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.082630  5009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:44.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.082650  5009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:44.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.083853  5009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:44.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.083905  5009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.65
07:56:44.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.085731  5009 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:44.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.65:0
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.65
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.5009
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.65
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/logs
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.085988  5009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.086333  5009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:44.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.091800  5014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.091807  5017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.091825  5015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.092434  5009 server_base.cc:1034] running on GCE node
07:56:44.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.092617  5009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:44.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.092810  5009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:44.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.093976  5009 hybrid_clock.cc:648] HybridClock initialized: now 1783497404093950 us; error 42 us; skew 500 ppm
07:56:44.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.095291  5009 webserver.cc:460] Webserver started at http://127.4.157.65:40963/ using document root <none> and password file <none>
07:56:44.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.095582  5009 fs_manager.cc:359] Metadata directory not provided
07:56:44.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.095670  5009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:44.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.095808  5009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
07:56:44.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.096885  5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/instance:
07:56:44.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "12bc83bf6ff640bc9820d875747f49cc"
07:56:44.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:44.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.097306  5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal/instance:
07:56:44.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "12bc83bf6ff640bc9820d875747f49cc"
07:56:44.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:44.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.098855  5009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
07:56:44.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.099785  5023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:44.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.099985  5009 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
07:56:44.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.100081  5009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:44.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "12bc83bf6ff640bc9820d875747f49cc"
07:56:44.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:44.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.100145  5009 fs_report.cc:383] FS layout report
07:56:44.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:44.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:44.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:44.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/data
07:56:44.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:44.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:44.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:44.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:44.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:44.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:44.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:44.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:44.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:44.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:44.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:44.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.130045  5009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:44.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.130429  5009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:44.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.130564  5009 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:44.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.130836  5009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:56:44.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.131269  5009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
07:56:44.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.131314  5009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
07:56:44.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.131338  5009 ts_tablet_manager.cc:610] Registered 0 tablets
07:56:44.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.131381  5009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
07:56:44.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.137352  5009 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.65:34783
07:56:44.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.137432  5136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.65:34783 every 8 connection(s)
07:56:44.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.137696  5009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:56:44.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.141685  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 5009
07:56:44.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.141835  4725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal/instance
07:56:44.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.143564  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:44.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:44.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:44.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data
07:56:44.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/logs
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.66:0
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.66
07:56:44.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.66
07:56:44.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:44.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:44.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:44.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:44.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:44.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:44.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.144650  5137 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:56:44.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.144714  5139 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:56:44.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.144747  5139 heartbeater.cc:464] Registering TS with master...
07:56:44.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.144747  5137 heartbeater.cc:464] Registering TS with master...
07:56:44.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.144964  5139 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:56:44.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.144964  5137 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:56:44.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.145498  4839 ts_manager.cc:194] Registered new tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:44.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.145530  4911 ts_manager.cc:194] Registered new tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:44.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.146165  4839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.65:38509
07:56:44.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.147035  5138 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:56:44.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.147116  5138 heartbeater.cc:464] Registering TS with master...
07:56:44.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.147183  5138 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:56:44.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.147522  4767 ts_manager.cc:194] Registered new tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:44.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.232589  5144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:44.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.232797  5144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:44.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.232818  5144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:44.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.233965  5144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:44.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.234018  5144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.66
07:56:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.235268  5144 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data
07:56:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.66:0
07:56:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.66
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.5144
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.66
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/logs
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.235448  5144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.235734  5144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:44.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.241080  5149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.241163  5152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.241165  5150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.241278  5144 server_base.cc:1034] running on GCE node
07:56:44.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.241788  5144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:44.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.241990  5144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:44.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.243156  5144 hybrid_clock.cc:648] HybridClock initialized: now 1783497404243142 us; error 26 us; skew 500 ppm
07:56:44.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.244588  5144 webserver.cc:460] Webserver started at http://127.4.157.66:36277/ using document root <none> and password file <none>
07:56:44.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.244899  5144 fs_manager.cc:359] Metadata directory not provided
07:56:44.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.244979  5144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:44.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.245136  5144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
07:56:44.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.246133  5144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/instance:
07:56:44.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "137d6728589143c1a8ae1e4cc280df77"
07:56:44.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:44.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.246528  5144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal/instance:
07:56:44.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "137d6728589143c1a8ae1e4cc280df77"
07:56:44.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:44.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.247969  5144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.001s
07:56:44.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.248769  5158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.248965  5144 fs_manager.cc:725] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.249122  5144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "137d6728589143c1a8ae1e4cc280df77"
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.249191  5144 fs_report.cc:383] FS layout report
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/data
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:44.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:44.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:44.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:44.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:44.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:44.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:44.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.275916  5144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:44.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.276293  5144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:44.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.276435  5144 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:44.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.276721  5144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:56:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.277195  5144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
07:56:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.277252  5144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
07:56:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.277287  5144 ts_tablet_manager.cc:610] Registered 0 tablets
07:56:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.277303  5144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
07:56:44.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.283995  5144 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.66:34353
07:56:44.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.284045  5271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.66:34353 every 8 connection(s)
07:56:44.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.284497  5144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:56:44.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.289209  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 5144
07:56:44.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.289325  4725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal/instance
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.291447  5274 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.291556  5274 heartbeater.cc:464] Registering TS with master...
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.291608  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/logs
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.67:0
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.67
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.67
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:44.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.291764  5274 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.291810  5272 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.291841  5272 heartbeater.cc:464] Registering TS with master...
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.291886  5272 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.291931  5273 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.291957  5273 heartbeater.cc:464] Registering TS with master...
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.292002  5273 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.292168  4839 ts_manager.cc:194] Registered new tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.292225  4911 ts_manager.cc:194] Registered new tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.292632  4839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.66:42675
07:56:44.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.293130  4767 ts_manager.cc:194] Registered new tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:56:44.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.373618  5279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:44.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.373831  5279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:44.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.373850  5279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:44.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.374938  5279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:44.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.374986  5279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.67
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.376243  5279 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.67:0
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.67
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:56:44.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.5279
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.67
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/logs
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.376487  5279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:44.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.376751  5279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:44.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.381817  5287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.381808  5284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:44.381862  5285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:44.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.382128  5279 server_base.cc:1034] running on GCE node
07:56:44.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.382300  5279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:44.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.382486  5279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:44.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.383642  5279 hybrid_clock.cc:648] HybridClock initialized: now 1783497404383608 us; error 44 us; skew 500 ppm
07:56:44.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.385190  5279 webserver.cc:460] Webserver started at http://127.4.157.67:46573/ using document root <none> and password file <none>
07:56:44.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.385479  5279 fs_manager.cc:359] Metadata directory not provided
07:56:44.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.385696  5279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:44.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.385977  5279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
07:56:44.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.387535  5279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/instance:
07:56:44.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f772946b4e6940dda07baac926a093bf"
07:56:44.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:44.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.388248  5279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal/instance:
07:56:44.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f772946b4e6940dda07baac926a093bf"
07:56:44.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:44.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.393043  5279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.002s
07:56:44.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.394074  5293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:44.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.394261  5279 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
07:56:44.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.394362  5279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:44.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f772946b4e6940dda07baac926a093bf"
07:56:44.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:44.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.394443  5279 fs_report.cc:383] FS layout report
07:56:44.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/data
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:44.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:44.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.421674  5279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:44.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.422066  5279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:44.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.422227  5279 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:44.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.422498  5279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:56:44.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.422909  5279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
07:56:44.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.422971  5279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
07:56:44.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.423005  5279 ts_tablet_manager.cc:610] Registered 0 tablets
07:56:44.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.423023  5279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
07:56:44.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.429580  5279 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.67:39629
07:56:44.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.429627  5406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.67:39629 every 8 connection(s)
07:56:44.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.430040  5279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:56:44.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.436373  5408 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:56:44.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.436494  5408 heartbeater.cc:464] Registering TS with master...
07:56:44.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.436379  5407 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:56:44.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.436398  5409 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:56:44.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.436555  5407 heartbeater.cc:464] Registering TS with master...
07:56:44.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.436564  5409 heartbeater.cc:464] Registering TS with master...
07:56:44.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.436689  5407 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:56:44.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.436689  5408 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:56:44.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.436689  5409 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:56:44.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.437106  4839 ts_manager.cc:194] Registered new tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:44.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.437166  4911 ts_manager.cc:194] Registered new tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:44.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.437328  4767 ts_manager.cc:194] Registered new tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:44.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.437561  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 5279
07:56:44.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.437572  4839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.67:54317
07:56:44.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.437638  4725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal/instance
07:56:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.440145  4725 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
07:56:44.441 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
07:56:44.442 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

07:56:44.451 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.4.157.126"
      port: 38135
    }
    bound_http_address {
      host: "127.4.157.126"
      port: 34573
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.4.157.125"
      port: 45997
    }
    bound_http_address {
      host: "127.4.157.125"
      port: 34215
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.4.157.124"
      port: 46883
    }
    bound_http_address {
      host: "127.4.157.124"
      port: 34775
    }
  }
}

07:56:44.457 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

07:56:44.459 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.4.157.65"
      port: 34783
    }
    bound_http_address {
      host: "127.4.157.65"
      port: 40963
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.4.157.66"
      port: 34353
    }
    bound_http_address {
      host: "127.4.157.66"
      port: 36277
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.4.157.67"
      port: 39629
    }
    bound_http_address {
      host: "127.4.157.67"
      port: 46573
    }
  }
}

07:56:44.459 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
07:56:44.644 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
07:56:44.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.797435  4981 catalog_manager.cc:1259] Loaded cluster ID: a42f5a84af0049a8aeb85a93fc647cad
07:56:44.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.797514  4981 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d: loading cluster ID for follower catalog manager: success
07:56:44.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.798804  4981 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d: acquiring CA information for follower catalog manager: success
07:56:44.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.799165  4981 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d06b7df478984ce3a473ce69c83a5f7d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
07:56:44.828 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.4.157.125:45997(127.4.157.125:45997)] connecting to peer
07:56:44.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.858978  4995 catalog_manager.cc:1259] Loaded cluster ID: a42f5a84af0049a8aeb85a93fc647cad
07:56:44.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.859055  4995 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05: loading cluster ID for follower catalog manager: success
07:56:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.860267  4995 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05: acquiring CA information for follower catalog manager: success
07:56:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:44.860627  4995 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a23c643f60244d58b7af6e724e7fbb05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
07:56:44.881 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.4.157.126:38135(127.4.157.126:38135)] connecting to peer
07:56:44.883 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.4.157.124:46883(127.4.157.124:46883)] connecting to peer
07:56:44.906 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.4.157.125:45997(127.4.157.125:45997)] Successfully connected to peer
07:56:44.906 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.4.157.126:38135(127.4.157.126:38135)] Successfully connected to peer
07:56:44.960 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.4.157.124:46883(127.4.157.124:46883)] Successfully connected to peer
07:56:44.980 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.4.157.125
07:56:44.980 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.4.157.126
07:56:45.070 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.4.157.124
07:56:45.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.147039  5139 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:56:45.193 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x11b97622, L:/127.0.0.1:55402 - R:/127.4.157.126:38135] using SASL/PLAIN
07:56:45.193 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xa0b890fa, L:/127.0.0.1:59046 - R:/127.4.157.125:45997] using SASL/PLAIN
07:56:45.200 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x2bc9cf0d, L:/127.0.0.1:59220 - R:/127.4.157.124:46883] using SASL/PLAIN
07:56:45.232 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
07:56:45.233 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=446), Traces: [0ms] refreshing cache from master, [29ms] Sub RPC ConnectToMaster: sending RPC to server master-127.4.157.125:45997, [90ms] Sub RPC ConnectToMaster: sending RPC to server master-127.4.157.126:38135, [91ms] Sub RPC ConnectToMaster: sending RPC to server master-127.4.157.124:46883, [418ms] Sub RPC ConnectToMaster: received response from server master-127.4.157.126:38135: OK, [418ms] Sub RPC ConnectToMaster: received response from server master-127.4.157.125:45997: OK, deferred=null) after lookup
07:56:45.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.284859  4839 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59046:
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776"
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
07:56:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:56:45.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
07:56:45.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:45.285255  4839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
07:56:45.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.293411  5274 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:56:45.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.298492  5341 tablet_service.cc:1467] Processing CreateTablet for tablet 85e494f8d75243559db1622f01fa36f7 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776 [id=097dc7b9706e4fd285b296f6a04a996c]), partition=RANGE (key) PARTITION UNBOUNDED
07:56:45.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.298573  5071 tablet_service.cc:1467] Processing CreateTablet for tablet 85e494f8d75243559db1622f01fa36f7 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776 [id=097dc7b9706e4fd285b296f6a04a996c]), partition=RANGE (key) PARTITION UNBOUNDED
07:56:45.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.298862  5341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e494f8d75243559db1622f01fa36f7. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:45.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.298933  5071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e494f8d75243559db1622f01fa36f7. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:45.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.302511  5438 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap starting.
07:56:45.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.302546  5206 tablet_service.cc:1467] Processing CreateTablet for tablet 85e494f8d75243559db1622f01fa36f7 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776 [id=097dc7b9706e4fd285b296f6a04a996c]), partition=RANGE (key) PARTITION UNBOUNDED
07:56:45.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.302871  5206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e494f8d75243559db1622f01fa36f7. 1 dirs total, 0 dirs full, 0 dirs failed
07:56:45.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.303207  5438 tablet_bootstrap.cc:654] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Neither blocks nor log segments found. Creating new log.
07:56:45.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.303540  5438 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Log is configured to *not* fsync() on all Append() calls
07:56:45.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.304189  5438 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: No bootstrap required, opened a new log
07:56:45.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.304375  5438 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
07:56:45.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.306741  5437 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Bootstrap starting.
07:56:45.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.307197  5438 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:45.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.307516  5438 raft_consensus.cc:374] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:45.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.307595  5437 tablet_bootstrap.cc:654] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Neither blocks nor log segments found. Creating new log.
07:56:45.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.307832  5438 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f772946b4e6940dda07baac926a093bf, State: Initialized, Role: FOLLOWER
07:56:45.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.307911  5437 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Log is configured to *not* fsync() on all Append() calls
07:56:45.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.308576  5438 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:45.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.308967  5441 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Bootstrap starting.
07:56:45.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.308991  5438 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
07:56:45.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.309940  5441 tablet_bootstrap.cc:654] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Neither blocks nor log segments found. Creating new log.
07:56:45.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.310109  5409 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:56:45.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.310766  5441 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Log is configured to *not* fsync() on all Append() calls
07:56:45.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.311570  5441 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: No bootstrap required, opened a new log
07:56:45.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.311648  5441 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
07:56:45.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.314177  5441 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:45.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.314473  5441 raft_consensus.cc:374] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:45.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.314532  5441 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137d6728589143c1a8ae1e4cc280df77, State: Initialized, Role: FOLLOWER
07:56:45.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.314667  5441 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:45.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.315034  5441 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
07:56:45.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.317613  5437 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: No bootstrap required, opened a new log
07:56:45.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.317817  5437 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.000s
07:56:45.320 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 097dc7b9706e4fd285b296f6a04a996c
07:56:45.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.320849  5437 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:45.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.321072  5437 raft_consensus.cc:374] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:56:45.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.321116  5437 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bc83bf6ff640bc9820d875747f49cc, State: Initialized, Role: FOLLOWER
07:56:45.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.321352  5437 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:45.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.321687  5437 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.004s
07:56:45.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:45.393244  5140 tablet.cc:2369] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:56:45.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:45.431262  5410 tablet.cc:2369] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:56:45.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.473249  5442 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:56:45.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.473456  5442 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:45.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.473910  5442 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:45.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.478514  5226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "f772946b4e6940dda07baac926a093bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137d6728589143c1a8ae1e4cc280df77" is_pre_election: true
07:56:45.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.478533  5091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "f772946b4e6940dda07baac926a093bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bc83bf6ff640bc9820d875747f49cc" is_pre_election: true
07:56:45.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.478706  5226 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f772946b4e6940dda07baac926a093bf in term 0.
07:56:45.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.478690  5091 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f772946b4e6940dda07baac926a093bf in term 0.
07:56:45.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.478994  5297 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, 137d6728589143c1a8ae1e4cc280df77, f772946b4e6940dda07baac926a093bf; no voters: 
07:56:45.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.479205  5442 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 0 FOLLOWER]: Leader pre-election won for term 1
07:56:45.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.479281  5442 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
07:56:45.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.479310  5442 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 0 FOLLOWER]: Advancing to term 1
07:56:45.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.480016  5442 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:45.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.480151  5442 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 1 election: Requested vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:45.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.480319  5091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "f772946b4e6940dda07baac926a093bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bc83bf6ff640bc9820d875747f49cc"
07:56:45.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.480321  5226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "f772946b4e6940dda07baac926a093bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137d6728589143c1a8ae1e4cc280df77"
07:56:45.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.480393  5091 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 0 FOLLOWER]: Advancing to term 1
07:56:45.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.480438  5226 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 0 FOLLOWER]: Advancing to term 1
07:56:45.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.481144  5091 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f772946b4e6940dda07baac926a093bf in term 1.
07:56:45.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.481338  5294 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, f772946b4e6940dda07baac926a093bf; no voters: 
07:56:45.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.481428  5226 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f772946b4e6940dda07baac926a093bf in term 1.
07:56:45.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.481438  5442 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 1 FOLLOWER]: Leader election won for term 1
07:56:45.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.481595  5442 raft_consensus.cc:686] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 1 LEADER]: Becoming Leader. State: Replica: f772946b4e6940dda07baac926a093bf, State: Running, Role: LEADER
07:56:45.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.481709  5442 consensus_queue.cc:237] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:45.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.482637  4839 catalog_manager.cc:5495] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f772946b4e6940dda07baac926a093bf (127.4.157.67). New cstate: current_term: 1 leader_uuid: "f772946b4e6940dda07baac926a093bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } health_report { overall_health: HEALTHY } } }
07:56:45.514 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 097dc7b9706e4fd285b296f6a04a996c
07:56:45.530 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:45.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:45.536547  5275 tablet.cc:2369] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:56:45.537 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299993, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:45.538 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=16), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [7ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=null) after lookup
07:56:45.548 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:45.549 [DEBUG - kudu-nio-1] (Connection.java:757) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] Successfully connected to peer
07:56:45.599 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x0e4c2125, L:/127.0.0.1:54914 - R:/127.4.157.67:39629] using TOKEN/null
07:56:45.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.601634  5226 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 1 FOLLOWER]: Refusing update from remote peer f772946b4e6940dda07baac926a093bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:56:45.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.601645  5091 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 1 FOLLOWER]: Refusing update from remote peer f772946b4e6940dda07baac926a093bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:56:45.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.602078  5442 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:56:45.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.602214  5448 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:56:45.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.604725  5451 mvcc.cc:205] Tried to move back new op lower bound from 7305205373341323264 to 7305205372853682176. Current Snapshot: MvccSnapshot[applied={T|T < 7305205373341323264}]
07:56:45.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.605477  5452 mvcc.cc:205] Tried to move back new op lower bound from 7305205373341323264 to 7305205372853682176. Current Snapshot: MvccSnapshot[applied={T|T < 7305205373341323264}]
07:56:45.673 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:45.675 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.157.67:39629
07:56:45.677 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:45.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:45.677475  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 5279
07:56:45.684 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:45.686 [DEBUG - kudu-nio-1] (Connection.java:239) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] handling channelInactive
07:56:45.687 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state READY due to: connection closed
07:56:45.687 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: connection closed
07:56:45.711 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:45.712 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:45.712 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [26ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@2128830519(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@36c4fb0f -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d31554c -> wakeup thread Time-limited test)) after lookup
07:56:45.713 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:45.714 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.714 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.715 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.731 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:45.731 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:45.732 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [26ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [28ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [30ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [30ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [46ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@2128830519(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@36c4fb0f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@128685651 after retry RPC@1911442964), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d31554c -> wakeup thread Time-limited test -> (continuation of Deferred@128685651 after retry RPC@1911442964))) after lookup
07:56:45.732 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:45.733 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.733 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.734 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.771 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:45.771 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:45.772 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=87), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [26ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [28ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [30ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [30ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [46ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [47ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [49ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [49ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [86ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@2128830519(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@36c4fb0f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@128685651 after retry RPC@1911442964) -> (continuation of Deferred@1756056612 after retry RPC@1807098315), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d31554c -> wakeup thread Time-limited test -> (continuation of Deferred@128685651 after retry RPC@1911442964) -> (continuation of Deferred@1756056612 after retry RPC@1807098315))) after lookup
07:56:45.773 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:45.774 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.774 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.774 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.871 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:45.872 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:45.872 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=187), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [26ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [28ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [30ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [30ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [46ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [47ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [49ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [49ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [86ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [88ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [89ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [89ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [186ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@2128830519(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@36c4fb0f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@128685651 after retry RPC@1911442964) -> (continuation of Deferred@1756056612 after retry RPC@1807098315) -> (continuation of Deferred@346782511 after retry RPC@1595394831), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d31554c -> wakeup thread Time-limited test -> (continuation of Deferred@128685651 after retry RPC@1911442964) -> (continuation of Deferred@1756056612 after retry RPC@1807098315) -> (continuation of Deferred@346782511 after retry RPC@1595394831))) after lookup
07:56:45.873 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:45.874 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.874 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.874 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.911 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:45.912 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:45.912 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=227), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [26ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [28ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [30ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [30ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [46ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [47ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [49ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [49ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [86ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [88ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [89ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [89ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [186ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [188ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [189ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [189ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [223ms] refreshing cache from master, [224ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [226ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@2128830519(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@36c4fb0f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@128685651 after retry RPC@1911442964) -> (continuation of Deferred@1756056612 after retry RPC@1807098315) -> (continuation of Deferred@346782511 after retry RPC@1595394831) -> (continuation of Deferred@1546820979 after retry RPC@1025276489), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d31554c -> wakeup thread Time-limited test -> (continuation of Deferred@128685651 after retry RPC@1911442964) -> (continuation of Deferred@1756056612 after retry RPC@1807098315) -> (continuation of Deferred@346782511 after retry RPC@1595394831) -> (continuation of Deferred@1546820979 after retry RPC@1025276489))) after lookup
07:56:45.913 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:45.914 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.914 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:45.915 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:47.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.127732  5461 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f772946b4e6940dda07baac926a093bf)
07:56:47.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.127830  5461 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:47.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.128137  5461 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:47.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.128484  5462 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f772946b4e6940dda07baac926a093bf)
07:56:47.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.128557  5462 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:47.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:47.128636  5162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:56:47.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.128822  5462 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:47.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:47.129166  5162 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:56:47.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:47.129387  5027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:56:47.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:47.129828  5027 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:56:47.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.131897  5091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "137d6728589143c1a8ae1e4cc280df77" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "12bc83bf6ff640bc9820d875747f49cc" is_pre_election: true
07:56:47.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.131989  5091 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137d6728589143c1a8ae1e4cc280df77 in term 1.
07:56:47.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.132193  5159 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, 137d6728589143c1a8ae1e4cc280df77; no voters: f772946b4e6940dda07baac926a093bf
07:56:47.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.132391  5461 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 1 FOLLOWER]: Leader pre-election won for term 2
07:56:47.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.132440  5461 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f772946b4e6940dda07baac926a093bf)
07:56:47.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.132465  5461 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 1 FOLLOWER]: Advancing to term 2
07:56:47.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.132541  5226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "12bc83bf6ff640bc9820d875747f49cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "137d6728589143c1a8ae1e4cc280df77" is_pre_election: true
07:56:47.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.133257  5461 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:47.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.133394  5226 raft_consensus.cc:2379] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 12bc83bf6ff640bc9820d875747f49cc in current term 2: Already voted for candidate 137d6728589143c1a8ae1e4cc280df77 in this term.
07:56:47.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.133416  5461 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 2 election: Requested vote from peers 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:47.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.133599  5091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "137d6728589143c1a8ae1e4cc280df77" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "12bc83bf6ff640bc9820d875747f49cc"
07:56:47.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.133688  5091 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 1 FOLLOWER]: Advancing to term 2
07:56:47.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.133608  5027 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc; no voters: 137d6728589143c1a8ae1e4cc280df77, f772946b4e6940dda07baac926a093bf
07:56:47.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:47.134207  5162 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:56:47.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.134397  5091 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137d6728589143c1a8ae1e4cc280df77 in term 2.
07:56:47.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.134477  5462 raft_consensus.cc:2726] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
07:56:47.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.134598  5159 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, 137d6728589143c1a8ae1e4cc280df77; no voters: f772946b4e6940dda07baac926a093bf
07:56:47.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.134686  5461 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 2 FOLLOWER]: Leader election won for term 2
07:56:47.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.134842  5461 raft_consensus.cc:686] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 2 LEADER]: Becoming Leader. State: Replica: 137d6728589143c1a8ae1e4cc280df77, State: Running, Role: LEADER
07:56:47.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.134933  5461 consensus_queue.cc:237] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:47.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.135720  4839 catalog_manager.cc:5495] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 reported cstate change: term changed from 1 to 2, leader changed from f772946b4e6940dda07baac926a093bf (127.4.157.67) to 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66). New cstate: current_term: 2 leader_uuid: "137d6728589143c1a8ae1e4cc280df77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } health_report { overall_health: UNKNOWN } } }
07:56:47.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.521453  5091 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 2 FOLLOWER]: Refusing update from remote peer 137d6728589143c1a8ae1e4cc280df77: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
07:56:47.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.521886  5461 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
07:56:47.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:47.524384  5162 consensus_peers.cc:487] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 -> Peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Couldn't send request to peer f772946b4e6940dda07baac926a093bf. Status: Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:47.891 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:47.892 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:47.893 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2207), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [26ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [28ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [30ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [30ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [46ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [47ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [49ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [49ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [86ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [88ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [89ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [89ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [186ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [188ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [189ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [189ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [223ms] refreshing cache from master, [224ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [226ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [228ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [230ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [230ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [2203ms] refreshing cache from master, [2203ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [2205ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@2128830519(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@36c4fb0f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@128685651 after retry RPC@1911442964) -> (continuation of Deferred@1756056612 after retry RPC@1807098315) -> (continuation of Deferred@346782511 after retry RPC@1595394831) -> (continuation of Deferred@1546820979 after retry RPC@1025276489) -> (continuation of Deferred@462580933 after retry RPC@598822974), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2d31554c -> wakeup thread Time-limited test -> (continuation of Deferred@128685651 after retry RPC@1911442964) -> (continuation of Deferred@1756056612 after retry RPC@1807098315) -> (continuation of Deferred@346782511 after retry RPC@1595394831) -> (continuation of Deferred@1546820979 after retry RPC@1025276489) -> (continuation of Deferred@462580933 after retry RPC@598822974))) after lookup
07:56:47.895 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:47.896 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] Successfully connected to peer
07:56:47.913 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xf208dfa3, L:/127.0.0.1:52148 - R:/127.4.157.66:34353] using TOKEN/null
07:56:47.925 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.157.67:39629
07:56:47.927 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:47.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:47.927579  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:47.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:47.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:47.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data
07:56:47.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:47.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/logs
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.67:39629
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.67
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46573
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.67
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:47.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:48.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.010324  5490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:48.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.010560  5490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:48.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.010592  5490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:48.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.011831  5490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:48.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.011911  5490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.67
07:56:48.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.013571  5490 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:48.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:48.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:48.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:48.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:48.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.67:39629
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.67
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46573
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.5490
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.67
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/logs
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:48.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:48.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:48.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:48.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:48.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:48.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:48.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:48.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:48.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.013866  5490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:48.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.014225  5490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:48.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.021137  5495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:48.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.021139  5496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:48.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.021257  5490 server_base.cc:1034] running on GCE node
07:56:48.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:48.021423  5498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:48.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.021667  5490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:48.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.021938  5490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:48.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.023120  5490 hybrid_clock.cc:648] HybridClock initialized: now 1783497408023091 us; error 46 us; skew 500 ppm
07:56:48.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.028487  5490 webserver.cc:460] Webserver started at http://127.4.157.67:46573/ using document root <none> and password file <none>
07:56:48.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.028807  5490 fs_manager.cc:359] Metadata directory not provided
07:56:48.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.028908  5490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:48.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.030519  5490 fs_manager.cc:709] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
07:56:48.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.031407  5504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:48.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.031618  5490 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.031750  5490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f772946b4e6940dda07baac926a093bf"
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.032223  5490 fs_report.cc:383] FS layout report
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/data
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:48.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:48.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:48.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.075721  5490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:48.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.076026  5490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:48.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.076128  5490 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:48.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.076327  5490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:56:48.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.076995  5511 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
07:56:48.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.078511  5490 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:48.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.078604  5490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s	user 0.000s	sys 0.000s
07:56:48.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.078645  5490 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
07:56:48.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.079311  5490 ts_tablet_manager.cc:610] Registered 1 tablets
07:56:48.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.079372  5490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
07:56:48.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.079412  5511 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap starting.
07:56:48.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.087071  5490 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.67:39629
07:56:48.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.087363  5618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.67:39629 every 8 connection(s)
07:56:48.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.087522  5490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:56:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.089118  5511 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Log is configured to *not* fsync() on all Append() calls
07:56:48.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.092051  5511 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:48.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.092391  5511 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap complete.
07:56:48.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.092842  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 5490
07:56:48.093 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:48.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.094543  5619 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:56:48.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.094642  5619 heartbeater.cc:464] Registering TS with master...
07:56:48.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.094837  5619 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:56:48.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.094548  5620 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:56:48.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.095561  5620 heartbeater.cc:464] Registering TS with master...
07:56:48.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.095623  5620 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:56:48.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.096055  4766 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:48.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.096756  5511 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.006s
07:56:48.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.097579  4911 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:48.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.098331  5511 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:48.098 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:48.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.098711  5511 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f772946b4e6940dda07baac926a093bf, State: Initialized, Role: FOLLOWER
07:56:48.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.098868  5511 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:48.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.099169  5511 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
07:56:48.102 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.157.66:34353
07:56:48.102 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:56:48.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.102555  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 5144
07:56:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.106045  5621 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:56:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.106105  5621 heartbeater.cc:464] Registering TS with master...
07:56:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.106174  5621 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:56:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.106581  4839 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:48.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.107125  4839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.67:40177
07:56:48.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:48.107942  5621 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:56:48.109 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] handling channelInactive
07:56:48.109 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state READY due to: connection closed
07:56:48.109 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:48.111 [DEBUG - Time-limited test] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:48.113 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.113 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.113 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.132 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:48.132 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:48.133 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=22), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [3ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1625040371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5164bafd -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@db31e29 -> wakeup thread Time-limited test)) after lookup
07:56:48.133 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:48.135 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.136 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.136 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.151 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:48.152 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:48.153 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [3ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [23ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [26ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [26ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1625040371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5164bafd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1104932836 after retry RPC@1410130848), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@db31e29 -> wakeup thread Time-limited test -> (continuation of Deferred@1104932836 after retry RPC@1410130848))) after lookup
07:56:48.153 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:48.154 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.154 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.154 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.172 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:48.172 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:48.173 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=62), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [3ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [23ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [26ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [26ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [44ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [44ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [58ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [61ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1625040371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5164bafd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1104932836 after retry RPC@1410130848) -> (continuation of Deferred@1348260627 after retry RPC@1259549249), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@db31e29 -> wakeup thread Time-limited test -> (continuation of Deferred@1104932836 after retry RPC@1410130848) -> (continuation of Deferred@1348260627 after retry RPC@1259549249))) after lookup
07:56:48.174 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:48.176 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.176 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.177 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.291 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:48.292 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:48.292 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=181), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [3ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [23ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [26ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [26ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [44ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [44ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [58ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [61ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [64ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [66ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [67ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [180ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1625040371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5164bafd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1104932836 after retry RPC@1410130848) -> (continuation of Deferred@1348260627 after retry RPC@1259549249) -> (continuation of Deferred@1504457254 after retry RPC@201298536), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@db31e29 -> wakeup thread Time-limited test -> (continuation of Deferred@1104932836 after retry RPC@1410130848) -> (continuation of Deferred@1348260627 after retry RPC@1259549249) -> (continuation of Deferred@1504457254 after retry RPC@201298536))) after lookup
07:56:48.293 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:48.294 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.295 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.295 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.712 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:48.712 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:48.713 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=602), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [3ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [23ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [26ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [26ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [44ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [44ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [58ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [61ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [64ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [66ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [67ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [180ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [183ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [185ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [186ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [599ms] refreshing cache from master, [599ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [601ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1625040371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5164bafd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1104932836 after retry RPC@1410130848) -> (continuation of Deferred@1348260627 after retry RPC@1259549249) -> (continuation of Deferred@1504457254 after retry RPC@201298536) -> (continuation of Deferred@1540018324 after retry RPC@1953403382), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@db31e29 -> wakeup thread Time-limited test -> (continuation of Deferred@1104932836 after retry RPC@1410130848) -> (continuation of Deferred@1348260627 after retry RPC@1259549249) -> (continuation of Deferred@1504457254 after retry RPC@201298536) -> (continuation of Deferred@1540018324 after retry RPC@1953403382))) after lookup
07:56:48.714 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:48.716 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.716 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:48.717 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.424829  5633 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 137d6728589143c1a8ae1e4cc280df77)
07:56:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.424923  5633 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.425117  5633 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:49.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:49.426002  5027 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353): Network error: Client connection negotiation failed: client connection to 127.4.157.66:34353: connect: Connection refused (error 111)
07:56:49.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.429375  5573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "12bc83bf6ff640bc9820d875747f49cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "f772946b4e6940dda07baac926a093bf" is_pre_election: true
07:56:49.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.429551  5573 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bc83bf6ff640bc9820d875747f49cc in term 1.
07:56:49.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.429808  5027 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, f772946b4e6940dda07baac926a093bf; no voters: 137d6728589143c1a8ae1e4cc280df77
07:56:49.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.429934  5633 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 2 FOLLOWER]: Leader pre-election won for term 3
07:56:49.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.429981  5633 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 2 FOLLOWER]: Starting leader election (detected failure of leader 137d6728589143c1a8ae1e4cc280df77)
07:56:49.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.430003  5633 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 2 FOLLOWER]: Advancing to term 3
07:56:49.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.430666  5633 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:49.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.430822  5633 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 3 election: Requested vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:49.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.431082  5573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "12bc83bf6ff640bc9820d875747f49cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "f772946b4e6940dda07baac926a093bf"
07:56:49.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.431190  5573 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 1 FOLLOWER]: Advancing to term 3
07:56:49.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:49.431357  5027 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353): Network error: Client connection negotiation failed: client connection to 127.4.157.66:34353: connect: Connection refused (error 111)
07:56:49.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.432363  5573 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bc83bf6ff640bc9820d875747f49cc in term 3.
07:56:49.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.432559  5027 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, f772946b4e6940dda07baac926a093bf; no voters: 137d6728589143c1a8ae1e4cc280df77
07:56:49.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.432669  5633 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 3 FOLLOWER]: Leader election won for term 3
07:56:49.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.432827  5633 raft_consensus.cc:686] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 3 LEADER]: Becoming Leader. State: Replica: 12bc83bf6ff640bc9820d875747f49cc, State: Running, Role: LEADER
07:56:49.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.433061  5633 consensus_queue.cc:237] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:49.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.433910  4839 catalog_manager.cc:5495] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc reported cstate change: term changed from 2 to 3, leader changed from 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66) to 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65). New cstate: current_term: 3 leader_uuid: "12bc83bf6ff640bc9820d875747f49cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } health_report { overall_health: UNKNOWN } } }
07:56:49.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:49.911998  5027 consensus_peers.cc:487] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc -> Peer 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353): Couldn't send request to peer 137d6728589143c1a8ae1e4cc280df77. Status: Network error: Client connection negotiation failed: client connection to 127.4.157.66:34353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:49.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.976014  5573 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 3 FOLLOWER]: Refusing update from remote peer 12bc83bf6ff640bc9820d875747f49cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
07:56:49.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:49.976446  5633 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
07:56:50.071 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:50.071 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:50.072 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1961), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [3ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [23ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [26ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [26ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [44ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [44ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [58ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [61ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [64ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [66ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [67ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [180ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [183ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [185ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [186ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [599ms] refreshing cache from master, [599ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [601ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [604ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [606ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [607ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [1958ms] refreshing cache from master, [1958ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [1960ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1625040371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5164bafd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1104932836 after retry RPC@1410130848) -> (continuation of Deferred@1348260627 after retry RPC@1259549249) -> (continuation of Deferred@1504457254 after retry RPC@201298536) -> (continuation of Deferred@1540018324 after retry RPC@1953403382) -> (continuation of Deferred@1394074887 after retry RPC@426819495), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@db31e29 -> wakeup thread Time-limited test -> (continuation of Deferred@1104932836 after retry RPC@1410130848) -> (continuation of Deferred@1348260627 after retry RPC@1259549249) -> (continuation of Deferred@1504457254 after retry RPC@201298536) -> (continuation of Deferred@1540018324 after retry RPC@1953403382) -> (continuation of Deferred@1394074887 after retry RPC@426819495))) after lookup
07:56:50.073 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:50.074 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] Successfully connected to peer
07:56:50.091 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x035c3ef7, L:/127.0.0.1:59680 - R:/127.4.157.65:34783] using TOKEN/null
07:56:50.102 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.157.66:34353
07:56:50.103 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.103989  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data
07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/logs
07:56:50.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.66:34353
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.66
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36277
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.66
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:50.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:50.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.188563  5662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:50.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.188773  5662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:50.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.188794  5662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:50.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.189924  5662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:50.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.189971  5662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.66
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.191237  5662 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.66:34353
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.66
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36277
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.5662
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.66
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/logs
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:50.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.191442  5662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.191702  5662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:50.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.197091  5668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:50.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.197186  5670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:50.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:50.197091  5667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:50.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.197386  5662 server_base.cc:1034] running on GCE node
07:56:50.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.197597  5662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:50.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.197826  5662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:50.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.198982  5662 hybrid_clock.cc:648] HybridClock initialized: now 1783497410198948 us; error 46 us; skew 500 ppm
07:56:50.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.200315  5662 webserver.cc:460] Webserver started at http://127.4.157.66:36277/ using document root <none> and password file <none>
07:56:50.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.200507  5662 fs_manager.cc:359] Metadata directory not provided
07:56:50.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.200562  5662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:50.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.202396  5662 fs_manager.cc:709] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
07:56:50.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.203061  5676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:50.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.203361  5662 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
07:56:50.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.203438  5662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:50.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "137d6728589143c1a8ae1e4cc280df77"
07:56:50.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.203815  5662 fs_report.cc:383] FS layout report
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/data
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:50.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.236617  5662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:50.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.236953  5662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:50.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.237119  5662 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:50.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.237344  5662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:56:50.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.237763  5683 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
07:56:50.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.239209  5662 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:50.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.239317  5662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s	user 0.000s	sys 0.000s
07:56:50.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.239360  5662 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
07:56:50.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.240016  5662 ts_tablet_manager.cc:610] Registered 1 tablets
07:56:50.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.240078  5662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
07:56:50.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.240187  5683 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Bootstrap starting.
07:56:50.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.247674  5662 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.66:34353
07:56:50.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.247761  5790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.66:34353 every 8 connection(s)
07:56:50.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.248147  5662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:56:50.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.249544  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 5662
07:56:50.250 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:50.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.249907  5683 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Log is configured to *not* fsync() on all Append() calls
07:56:50.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.253499  5683 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:50.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.253816  5683 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Bootstrap complete.
07:56:50.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.254943  5683 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Time spent bootstrapping tablet: real 0.015s	user 0.006s	sys 0.006s
07:56:50.255 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:50.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.256912  5792 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:56:50.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.257145  5792 heartbeater.cc:464] Registering TS with master...
07:56:50.257 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.157.65:34783
07:56:50.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.257447  5792 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:56:50.257 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:56:50.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.257884  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 5009
07:56:50.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.258565  4767 ts_manager.cc:194] Re-registered known tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:56:50.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.260417  5683 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:50.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.261193  5683 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137d6728589143c1a8ae1e4cc280df77, State: Initialized, Role: FOLLOWER
07:56:50.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.261327  5683 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:50.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.261606  5683 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
07:56:50.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.262394  5793 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:56:50.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.262441  5793 heartbeater.cc:464] Registering TS with master...
07:56:50.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.262501  5793 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:56:50.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.262782  4839 ts_manager.cc:194] Re-registered known tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:56:50.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.263356  4839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.66:36719
07:56:50.264 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] handling channelInactive
07:56:50.264 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state READY due to: connection closed
07:56:50.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.263839  5791 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:56:50.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.263948  5791 heartbeater.cc:464] Registering TS with master...
07:56:50.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.264052  5791 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:56:50.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.264134  5793 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:56:50.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:50.264325  4911 ts_manager.cc:194] Re-registered known tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:56:50.265 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:50.266 [DEBUG - Time-limited test] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:50.269 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.269 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.269 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.292 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:50.292 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:50.293 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [3ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [25ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1430618607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5c79fb9b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4170ef0d -> wakeup thread Time-limited test)) after lookup
07:56:50.294 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:50.295 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.295 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.296 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.312 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:50.312 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:50.313 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [3ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [25ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [28ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [29ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [30ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [45ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1430618607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5c79fb9b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@34383649 after retry RPC@919569921), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4170ef0d -> wakeup thread Time-limited test -> (continuation of Deferred@34383649 after retry RPC@919569921))) after lookup
07:56:50.313 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:50.316 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.316 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.316 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.353 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:50.353 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:50.354 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=88), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [3ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [25ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [28ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [29ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [30ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [45ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [47ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [50ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [50ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [82ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [86ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1430618607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5c79fb9b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@34383649 after retry RPC@919569921) -> (continuation of Deferred@1669940633 after retry RPC@691808359), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4170ef0d -> wakeup thread Time-limited test -> (continuation of Deferred@34383649 after retry RPC@919569921) -> (continuation of Deferred@1669940633 after retry RPC@691808359))) after lookup
07:56:50.354 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:50.355 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.356 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.356 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.392 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:50.392 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:50.393 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=127), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [3ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [25ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [28ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [29ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [30ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [45ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [47ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [50ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [50ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [82ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [86ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [88ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [90ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [90ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [123ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [125ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1430618607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5c79fb9b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@34383649 after retry RPC@919569921) -> (continuation of Deferred@1669940633 after retry RPC@691808359) -> (continuation of Deferred@496400569 after retry RPC@2062225168), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4170ef0d -> wakeup thread Time-limited test -> (continuation of Deferred@34383649 after retry RPC@919569921) -> (continuation of Deferred@1669940633 after retry RPC@691808359) -> (continuation of Deferred@496400569 after retry RPC@2062225168))) after lookup
07:56:50.394 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:50.396 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.396 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.396 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.871 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:50.871 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:50.872 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=606), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [3ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [25ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [28ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [29ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [30ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [45ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [47ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [50ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [50ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [82ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [86ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [88ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [90ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [90ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [123ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [125ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [127ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [130ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [130ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [602ms] refreshing cache from master, [602ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [604ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1430618607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5c79fb9b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@34383649 after retry RPC@919569921) -> (continuation of Deferred@1669940633 after retry RPC@691808359) -> (continuation of Deferred@496400569 after retry RPC@2062225168) -> (continuation of Deferred@665671517 after retry RPC@94129975), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4170ef0d -> wakeup thread Time-limited test -> (continuation of Deferred@34383649 after retry RPC@919569921) -> (continuation of Deferred@1669940633 after retry RPC@691808359) -> (continuation of Deferred@496400569 after retry RPC@2062225168) -> (continuation of Deferred@665671517 after retry RPC@94129975))) after lookup
07:56:50.873 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:50.874 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.874 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:50.874 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:51.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.602753  5804 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 12bc83bf6ff640bc9820d875747f49cc)
07:56:51.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.602855  5804 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:51.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.603210  5804 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:51.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.603647  5505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111)
07:56:51.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.604144  5505 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783): Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111)
07:56:51.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.607364  5745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "f772946b4e6940dda07baac926a093bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "137d6728589143c1a8ae1e4cc280df77" is_pre_election: true
07:56:51.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.607555  5745 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f772946b4e6940dda07baac926a093bf in term 2.
07:56:51.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.607915  5508 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 137d6728589143c1a8ae1e4cc280df77, f772946b4e6940dda07baac926a093bf; no voters: 12bc83bf6ff640bc9820d875747f49cc
07:56:51.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.608106  5804 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 3 FOLLOWER]: Leader pre-election won for term 4
07:56:51.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.608197  5804 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 3 FOLLOWER]: Starting leader election (detected failure of leader 12bc83bf6ff640bc9820d875747f49cc)
07:56:51.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.608215  5804 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 3 FOLLOWER]: Advancing to term 4
07:56:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.608866  5804 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.609054  5804 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 4 election: Requested vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.609236  5745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "f772946b4e6940dda07baac926a093bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "137d6728589143c1a8ae1e4cc280df77"
07:56:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.609378  5745 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 2 FOLLOWER]: Advancing to term 4
07:56:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.609666  5505 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783): Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111)
07:56:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.610664  5745 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f772946b4e6940dda07baac926a093bf in term 4.
07:56:51.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.610910  5508 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 137d6728589143c1a8ae1e4cc280df77, f772946b4e6940dda07baac926a093bf; no voters: 12bc83bf6ff640bc9820d875747f49cc
07:56:51.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.611034  5804 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 4 FOLLOWER]: Leader election won for term 4
07:56:51.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.611174  5804 raft_consensus.cc:686] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 4 LEADER]: Becoming Leader. State: Replica: f772946b4e6940dda07baac926a093bf, State: Running, Role: LEADER
07:56:51.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.611321  5804 consensus_queue.cc:237] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:51.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.612089  4839 catalog_manager.cc:5495] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf reported cstate change: term changed from 3 to 4, leader changed from 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65) to f772946b4e6940dda07baac926a093bf (127.4.157.67). New cstate: current_term: 4 leader_uuid: "f772946b4e6940dda07baac926a093bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } health_report { overall_health: HEALTHY } } }
07:56:51.812 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:51.812 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:51.813 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1547), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [3ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [25ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [28ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [29ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [30ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [45ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [47ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [50ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [50ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [82ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [86ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [88ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [90ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [90ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [123ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [125ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [127ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [130ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [130ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [602ms] refreshing cache from master, [602ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [604ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [606ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [608ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [608ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [1543ms] refreshing cache from master, [1543ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [1545ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1430618607(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5c79fb9b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@34383649 after retry RPC@919569921) -> (continuation of Deferred@1669940633 after retry RPC@691808359) -> (continuation of Deferred@496400569 after retry RPC@2062225168) -> (continuation of Deferred@665671517 after retry RPC@94129975) -> (continuation of Deferred@1820006161 after retry RPC@1052508783), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4170ef0d -> wakeup thread Time-limited test -> (continuation of Deferred@34383649 after retry RPC@919569921) -> (continuation of Deferred@1669940633 after retry RPC@691808359) -> (continuation of Deferred@496400569 after retry RPC@2062225168) -> (continuation of Deferred@665671517 after retry RPC@94129975) -> (continuation of Deferred@1820006161 after retry RPC@1052508783))) after lookup
07:56:51.814 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:51.816 [DEBUG - kudu-nio-1] (Connection.java:757) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] Successfully connected to peer
07:56:51.836 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x1a7a9254, L:/127.0.0.1:38784 - R:/127.4.157.67:39629] using TOKEN/null
07:56:51.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.838516  5745 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 4 FOLLOWER]: Refusing update from remote peer f772946b4e6940dda07baac926a093bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
07:56:51.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.838632  5505 consensus_peers.cc:487] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf -> Peer 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783): Couldn't send request to peer 12bc83bf6ff640bc9820d875747f49cc. Status: Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:51.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.838876  5804 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
07:56:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.841177  5823 mvcc.cc:205] Tried to move back new op lower bound from 7305205398887690240 to 7305205397960626176. Current Snapshot: MvccSnapshot[applied={T|T < 7305205398887690240}]
07:56:51.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.842348  5825 mvcc.cc:205] Tried to move back new op lower bound from 7305205398887690240 to 7305205397960626176. Current Snapshot: MvccSnapshot[applied={T|T < 7305205398887690240}]
07:56:51.852 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.157.65:34783
07:56:51.852 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:56:51.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.852636  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/logs
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.65:34783
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.65
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40963
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.65
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:51.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.934222  5832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:51.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.934427  5832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:51.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.934455  5832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:51.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.935561  5832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:51.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.935606  5832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.65
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.937070  5832 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.65:34783
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.65
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40963
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.5832
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.65
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/logs
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:51.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:51.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:51.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:51.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:51.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:51.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:51.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.937271  5832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:51.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.937517  5832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:51.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.943442  5832 server_base.cc:1034] running on GCE node
07:56:51.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.943229  5837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:51.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.943450  5840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:51.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:51.943231  5838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:51.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.944310  5832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:51.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.944623  5832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:51.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.945858  5832 hybrid_clock.cc:648] HybridClock initialized: now 1783497411945841 us; error 29 us; skew 500 ppm
07:56:51.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.947654  5832 webserver.cc:460] Webserver started at http://127.4.157.65:40963/ using document root <none> and password file <none>
07:56:51.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.947985  5832 fs_manager.cc:359] Metadata directory not provided
07:56:51.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.948077  5832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:51.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.949870  5832 fs_manager.cc:709] Time spent opening directory manager: real 0.001s	user 0.000s	sys 0.002s
07:56:51.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.950727  5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:51.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.950994  5832 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
07:56:51.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.951110  5832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:51.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "12bc83bf6ff640bc9820d875747f49cc"
07:56:51.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:51.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.951521  5832 fs_report.cc:383] FS layout report
07:56:51.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:51.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:51.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:51.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/data
07:56:51.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:51.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:51.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:51.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:51.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:51.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:51.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:51.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:51.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:51.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:51.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:51.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.965889  5832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:51.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.966310  5832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:51.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.966487  5832 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:51.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.966763  5832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:56:51.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.967413  5853 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
07:56:51.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.968664  5832 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:51.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.968748  5832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s	user 0.000s	sys 0.000s
07:56:51.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.968787  5832 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
07:56:51.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.969538  5832 ts_tablet_manager.cc:610] Registered 1 tablets
07:56:51.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.969597  5832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
07:56:51.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.969653  5853 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Bootstrap starting.
07:56:51.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.976768  5832 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.65:34783
07:56:51.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.976848  5960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.65:34783 every 8 connection(s)
07:56:51.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.977203  5832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:56:51.978 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:51.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.977586  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 5832
07:56:51.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.979180  5853 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Log is configured to *not* fsync() on all Append() calls
07:56:51.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.982558  5853 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:51.982 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:51.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.982872  5853 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Bootstrap complete.
07:56:51.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.984050  5853 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Time spent bootstrapping tablet: real 0.014s	user 0.001s	sys 0.009s
07:56:51.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.985715  5853 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:51.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.986335  5853 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bc83bf6ff640bc9820d875747f49cc, State: Initialized, Role: FOLLOWER
07:56:51.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.986502  5853 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:51.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.986779  5853 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Time spent starting tablet: real 0.003s	user 0.005s	sys 0.000s
07:56:51.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.986801  5962 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:56:51.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.986891  5962 heartbeater.cc:464] Registering TS with master...
07:56:51.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.987066  5962 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:56:51.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.987618  4767 ts_manager.cc:194] Re-registered known tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:51.989 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.157.67:39629
07:56:51.989 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:51.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.989837  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 5490
07:56:51.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.991425  5961 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:56:51.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.991482  5961 heartbeater.cc:464] Registering TS with master...
07:56:51.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.991550  5961 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:56:51.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.991880  4911 ts_manager.cc:194] Re-registered known tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:51.996 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:51.996 [DEBUG - kudu-nio-1] (Connection.java:239) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] handling channelInactive
07:56:51.996 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state READY due to: connection closed
07:56:51.997 [DEBUG - Time-limited test] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:51.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.999274  5963 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:56:51.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.999359  5963 heartbeater.cc:464] Registering TS with master...
07:56:51.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.999431  5963 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:56:51.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:51.999709  4839 ts_manager.cc:194] Re-registered known tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:52.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.000247  4839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.65:60177
07:56:52.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:52.001277  5963 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:56:52.001 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.001 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.002 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.011 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:52.011 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:52.012 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [6ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [6ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@206272410(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@7b4d0fc3 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21fa6d7 -> wakeup thread Time-limited test)) after lookup
07:56:52.012 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:52.014 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.015 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.015 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.032 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:52.032 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:52.032 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [6ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [6ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [16ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [19ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [19ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@206272410(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@7b4d0fc3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21fa6d7 -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653))) after lookup
07:56:52.033 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:52.033 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.034 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.034 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.072 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:52.072 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:52.072 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=76), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [6ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [6ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [16ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [19ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [19ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [36ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [38ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [38ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [75ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@206272410(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@7b4d0fc3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21fa6d7 -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810))) after lookup
07:56:52.073 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:52.075 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.075 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.075 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.191 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:52.191 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:52.192 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=195), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [6ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [6ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [16ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [19ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [19ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [36ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [38ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [38ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [75ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [76ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [79ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [79ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [192ms] refreshing cache from master, [192ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [194ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@206272410(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@7b4d0fc3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810) -> (continuation of Deferred@1248579528 after retry RPC@2065217563), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21fa6d7 -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810) -> (continuation of Deferred@1248579528 after retry RPC@2065217563))) after lookup
07:56:52.193 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:52.193 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.194 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.194 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.331 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:52.331 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:52.332 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=335), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [6ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [6ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [16ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [19ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [19ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [36ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [38ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [38ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [75ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [76ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [79ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [79ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [192ms] refreshing cache from master, [192ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [194ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [196ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [198ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [198ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [332ms] refreshing cache from master, [332ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [334ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@206272410(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@7b4d0fc3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810) -> (continuation of Deferred@1248579528 after retry RPC@2065217563) -> (continuation of Deferred@477864165 after retry RPC@2029786360), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21fa6d7 -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810) -> (continuation of Deferred@1248579528 after retry RPC@2065217563) -> (continuation of Deferred@477864165 after retry RPC@2029786360))) after lookup
07:56:52.333 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:52.334 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.334 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.335 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.891 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:52.891 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:52.892 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=895), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [6ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [6ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [16ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [19ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [19ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [36ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [38ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [38ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [75ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [76ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [79ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [79ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [192ms] refreshing cache from master, [192ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [194ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [196ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [198ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [198ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [332ms] refreshing cache from master, [332ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [334ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [336ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [338ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [339ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [892ms] refreshing cache from master, [892ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [894ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@206272410(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@7b4d0fc3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810) -> (continuation of Deferred@1248579528 after retry RPC@2065217563) -> (continuation of Deferred@477864165 after retry RPC@2029786360) -> (continuation of Deferred@850699356 after retry RPC@712453414), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21fa6d7 -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810) -> (continuation of Deferred@1248579528 after retry RPC@2065217563) -> (continuation of Deferred@477864165 after retry RPC@2029786360) -> (continuation of Deferred@850699356 after retry RPC@712453414))) after lookup
07:56:52.893 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:52.894 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.894 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.894 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.951 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:52.952 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:52.952 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=956), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [6ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [6ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [16ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [19ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [19ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [36ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [38ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [38ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [75ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [76ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [79ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [79ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [192ms] refreshing cache from master, [192ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [194ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [196ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [198ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [198ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [332ms] refreshing cache from master, [332ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [334ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [336ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [338ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [339ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [892ms] refreshing cache from master, [892ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [894ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [896ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [898ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [898ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [952ms] refreshing cache from master, [953ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [954ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@206272410(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@7b4d0fc3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810) -> (continuation of Deferred@1248579528 after retry RPC@2065217563) -> (continuation of Deferred@477864165 after retry RPC@2029786360) -> (continuation of Deferred@850699356 after retry RPC@712453414) -> (continuation of Deferred@1126745853 after retry RPC@1106475383), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21fa6d7 -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810) -> (continuation of Deferred@1248579528 after retry RPC@2065217563) -> (continuation of Deferred@477864165 after retry RPC@2029786360) -> (continuation of Deferred@850699356 after retry RPC@712453414) -> (continuation of Deferred@1126745853 after retry RPC@1106475383))) after lookup
07:56:52.953 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:52.955 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.955 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:52.956 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:53.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.228739  5974 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:56:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.228860  5974 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.229245  5974 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:53.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.229959  5850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:56:53.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.230475  5850 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:56:53.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.233023  5745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "12bc83bf6ff640bc9820d875747f49cc" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "137d6728589143c1a8ae1e4cc280df77" is_pre_election: true
07:56:53.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.233418  5850 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc; no voters: 137d6728589143c1a8ae1e4cc280df77, f772946b4e6940dda07baac926a093bf
07:56:53.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.233574  5974 raft_consensus.cc:2726] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
07:56:53.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.351039  5979 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f772946b4e6940dda07baac926a093bf)
07:56:53.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.351133  5979 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:53.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.351480  5979 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:53.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.351955  5680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:56:53.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.352566  5680 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:56:53.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.355659  5915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "137d6728589143c1a8ae1e4cc280df77" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "12bc83bf6ff640bc9820d875747f49cc" is_pre_election: true
07:56:53.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.355772  5915 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137d6728589143c1a8ae1e4cc280df77 in term 3.
07:56:53.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.355988  5677 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, 137d6728589143c1a8ae1e4cc280df77; no voters: f772946b4e6940dda07baac926a093bf
07:56:53.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.356228  5979 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 4 FOLLOWER]: Leader pre-election won for term 5
07:56:53.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.356349  5979 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f772946b4e6940dda07baac926a093bf)
07:56:53.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.356427  5979 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 4 FOLLOWER]: Advancing to term 5
07:56:53.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.357219  5979 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:53.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.357405  5979 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 5 election: Requested vote from peers 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:53.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.357566  5915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "137d6728589143c1a8ae1e4cc280df77" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "12bc83bf6ff640bc9820d875747f49cc"
07:56:53.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.357643  5915 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 3 FOLLOWER]: Advancing to term 5
07:56:53.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.357810  5680 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:56:53.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.358806  5915 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137d6728589143c1a8ae1e4cc280df77 in term 5.
07:56:53.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.359037  5677 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, 137d6728589143c1a8ae1e4cc280df77; no voters: f772946b4e6940dda07baac926a093bf
07:56:53.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.359213  5979 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 5 FOLLOWER]: Leader election won for term 5
07:56:53.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.359432  5979 raft_consensus.cc:686] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 5 LEADER]: Becoming Leader. State: Replica: 137d6728589143c1a8ae1e4cc280df77, State: Running, Role: LEADER
07:56:53.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.359527  5979 consensus_queue.cc:237] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:53.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.360343  4839 catalog_manager.cc:5495] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 reported cstate change: term changed from 4 to 5, leader changed from f772946b4e6940dda07baac926a093bf (127.4.157.67) to 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66). New cstate: current_term: 5 leader_uuid: "137d6728589143c1a8ae1e4cc280df77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } health_report { overall_health: UNKNOWN } } }
07:56:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:53.922796  5680 consensus_peers.cc:487] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 -> Peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Couldn't send request to peer f772946b4e6940dda07baac926a093bf. Status: Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:53.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.931047  5915 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 5 FOLLOWER]: Refusing update from remote peer 137d6728589143c1a8ae1e4cc280df77: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
07:56:53.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:53.931486  5997 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
07:56:54.991 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:54.992 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:54.992 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=16, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=2995), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [6ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [6ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [16ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [19ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [19ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [36ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [38ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [38ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [75ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [76ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [79ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [79ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [192ms] refreshing cache from master, [192ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [194ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [196ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [198ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [198ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [332ms] refreshing cache from master, [332ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [334ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [336ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [338ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [339ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [892ms] refreshing cache from master, [892ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [894ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [896ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [898ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [898ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [952ms] refreshing cache from master, [953ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [954ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [957ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [960ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [960ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [2992ms] refreshing cache from master, [2993ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [2994ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@206272410(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@7b4d0fc3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810) -> (continuation of Deferred@1248579528 after retry RPC@2065217563) -> (continuation of Deferred@477864165 after retry RPC@2029786360) -> (continuation of Deferred@850699356 after retry RPC@712453414) -> (continuation of Deferred@1126745853 after retry RPC@1106475383) -> (continuation of Deferred@1064868437 after retry RPC@1704616972), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21fa6d7 -> wakeup thread Time-limited test -> (continuation of Deferred@1874422011 after retry RPC@284115653) -> (continuation of Deferred@1876915031 after retry RPC@673397810) -> (continuation of Deferred@1248579528 after retry RPC@2065217563) -> (continuation of Deferred@477864165 after retry RPC@2029786360) -> (continuation of Deferred@850699356 after retry RPC@712453414) -> (continuation of Deferred@1126745853 after retry RPC@1106475383) -> (continuation of Deferred@1064868437 after retry RPC@1704616972))) after lookup
07:56:54.993 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:54.994 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] Successfully connected to peer
07:56:55.018 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xb5c0eca7, L:/127.0.0.1:40300 - R:/127.4.157.66:34353] using TOKEN/null
07:56:55.034 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.157.67:39629
07:56:55.034 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.034915  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/logs
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:55.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.67:39629
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.67
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46573
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.67
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:55.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.122717  6020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:55.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.123018  6020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:55.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.123050  6020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.124402  6020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.124495  6020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.67
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.126013  6020 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.67:39629
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.67
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46573
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.6020
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.67
07:56:55.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/logs
07:56:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.126293  6020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.126652  6020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:55.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.132045  6027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:55.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.132076  6028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:55.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:55.132295  6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:55.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.132942  6020 server_base.cc:1034] running on GCE node
07:56:55.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.133251  6020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:55.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.133565  6020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:55.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.134709  6020 hybrid_clock.cc:648] HybridClock initialized: now 1783497415134682 us; error 44 us; skew 500 ppm
07:56:55.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.140913  6020 webserver.cc:460] Webserver started at http://127.4.157.67:46573/ using document root <none> and password file <none>
07:56:55.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.141212  6020 fs_manager.cc:359] Metadata directory not provided
07:56:55.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.141294  6020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:55.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.142690  6020 fs_manager.cc:709] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
07:56:55.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.143404  6036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:55.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.143594  6020 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.143702  6020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f772946b4e6940dda07baac926a093bf"
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.144137  6020 fs_report.cc:383] FS layout report
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/data
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:55.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:55.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.155224  6020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:55.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.155652  6020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:55.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.155809  6020 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:55.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.156050  6020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:56:55.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.156554  6043 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
07:56:55.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.158339  6020 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:55.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.158396  6020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s	user 0.000s	sys 0.000s
07:56:55.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.158430  6020 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
07:56:55.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.159230  6020 ts_tablet_manager.cc:610] Registered 1 tablets
07:56:55.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.159272  6020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
07:56:55.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.159400  6043 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap starting.
07:56:55.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.165985  6020 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.67:39629
07:56:55.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.166386  6020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:56:55.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.169265  6043 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Log is configured to *not* fsync() on all Append() calls
07:56:55.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.173039  6043 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:55.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.173358  6043 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap complete.
07:56:55.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.173797  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 6020
07:56:55.174 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:55.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.174475  6043 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.006s
07:56:55.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.175899  6152 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:56:55.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.175978  6152 heartbeater.cc:464] Registering TS with master...
07:56:55.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.175846  6043 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:55.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.176177  6152 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:56:55.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.176164  6043 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: f772946b4e6940dda07baac926a093bf, State: Initialized, Role: FOLLOWER
07:56:55.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.176328  6043 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:55.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.176608  6043 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
07:56:55.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.177095  6150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.67:39629 every 8 connection(s)
07:56:55.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.178416  4767 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:55.180 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:55.182 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.157.66:34353
07:56:55.182 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:56:55.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.182600  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 5662
07:56:55.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.184016  6151 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:56:55.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.184074  6151 heartbeater.cc:464] Registering TS with master...
07:56:55.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.184140  6151 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:56:55.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.184453  4911 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:55.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.185062  6153 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:56:55.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.185209  6153 heartbeater.cc:464] Registering TS with master...
07:56:55.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.185340  6153 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:56:55.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.186892  4838 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:55.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.187414  4838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.67:42503
07:56:55.188 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] handling channelInactive
07:56:55.188 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:55.188 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state READY due to: connection closed
07:56:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:55.188354  6153 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:56:55.189 [DEBUG - Time-limited test] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:55.191 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.191 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.192 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.211 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:55.211 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:55.212 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1995529524(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@1c512b67 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3fa81a -> wakeup thread Time-limited test)) after lookup
07:56:55.213 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:55.214 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.214 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.214 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.231 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:55.232 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:55.232 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [24ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [25ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [25ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [41ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1995529524(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@1c512b67 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@731484255 after retry RPC@2003494086), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3fa81a -> wakeup thread Time-limited test -> (continuation of Deferred@731484255 after retry RPC@2003494086))) after lookup
07:56:55.233 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:55.234 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.235 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.235 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.271 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:55.271 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:55.292 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [24ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [25ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [25ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [41ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [46ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [46ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [81ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1995529524(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@1c512b67 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@731484255 after retry RPC@2003494086) -> (continuation of Deferred@1319156764 after retry RPC@1858874842), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3fa81a -> wakeup thread Time-limited test -> (continuation of Deferred@731484255 after retry RPC@2003494086) -> (continuation of Deferred@1319156764 after retry RPC@1858874842))) after lookup
07:56:55.294 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:55.295 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.295 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.295 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.411 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:55.412 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:55.412 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=223), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [24ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [25ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [25ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [41ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [46ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [46ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [81ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [104ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [106ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [106ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [220ms] refreshing cache from master, [220ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [222ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1995529524(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@1c512b67 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@731484255 after retry RPC@2003494086) -> (continuation of Deferred@1319156764 after retry RPC@1858874842) -> (continuation of Deferred@1772116630 after retry RPC@172866412), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3fa81a -> wakeup thread Time-limited test -> (continuation of Deferred@731484255 after retry RPC@2003494086) -> (continuation of Deferred@1319156764 after retry RPC@1858874842) -> (continuation of Deferred@1772116630 after retry RPC@172866412))) after lookup
07:56:55.412 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:55.415 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.415 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.415 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.890 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:55.891 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:55.891 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=702), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [24ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [25ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [25ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [41ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [46ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [46ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [81ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [104ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [106ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [106ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [220ms] refreshing cache from master, [220ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [222ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [223ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [226ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [226ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [699ms] refreshing cache from master, [699ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [701ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1995529524(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@1c512b67 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@731484255 after retry RPC@2003494086) -> (continuation of Deferred@1319156764 after retry RPC@1858874842) -> (continuation of Deferred@1772116630 after retry RPC@172866412) -> (continuation of Deferred@78892860 after retry RPC@1665307165), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3fa81a -> wakeup thread Time-limited test -> (continuation of Deferred@731484255 after retry RPC@2003494086) -> (continuation of Deferred@1319156764 after retry RPC@1858874842) -> (continuation of Deferred@1772116630 after retry RPC@172866412) -> (continuation of Deferred@78892860 after retry RPC@1665307165))) after lookup
07:56:55.892 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:56:55.893 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.893 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:55.894 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:56:56.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.533281  6166 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 137d6728589143c1a8ae1e4cc280df77)
07:56:56.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.533418  6166 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:56.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.533587  6166 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:56.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:56.534513  5850 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353): Network error: Client connection negotiation failed: client connection to 127.4.157.66:34353: connect: Connection refused (error 111)
07:56:56.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.537681  6092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "12bc83bf6ff640bc9820d875747f49cc" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "f772946b4e6940dda07baac926a093bf" is_pre_election: true
07:56:56.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.537842  6092 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bc83bf6ff640bc9820d875747f49cc in term 4.
07:56:56.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.538147  5850 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, f772946b4e6940dda07baac926a093bf; no voters: 137d6728589143c1a8ae1e4cc280df77
07:56:56.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.538280  6166 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 5 FOLLOWER]: Leader pre-election won for term 6
07:56:56.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.538379  6166 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 5 FOLLOWER]: Starting leader election (detected failure of leader 137d6728589143c1a8ae1e4cc280df77)
07:56:56.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.538408  6166 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 5 FOLLOWER]: Advancing to term 6
07:56:56.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.539069  6166 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:56.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.539238  6166 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 6 election: Requested vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:56:56.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.539589  6092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "12bc83bf6ff640bc9820d875747f49cc" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "f772946b4e6940dda07baac926a093bf"
07:56:56.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.539700  6092 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 4 FOLLOWER]: Advancing to term 6
07:56:56.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:56.539939  5850 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353): Network error: Client connection negotiation failed: client connection to 127.4.157.66:34353: connect: Connection refused (error 111)
07:56:56.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.540904  6092 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bc83bf6ff640bc9820d875747f49cc in term 6.
07:56:56.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.541105  5850 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, f772946b4e6940dda07baac926a093bf; no voters: 137d6728589143c1a8ae1e4cc280df77
07:56:56.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.541194  6166 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 6 FOLLOWER]: Leader election won for term 6
07:56:56.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.541355  6166 raft_consensus.cc:686] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 6 LEADER]: Becoming Leader. State: Replica: 12bc83bf6ff640bc9820d875747f49cc, State: Running, Role: LEADER
07:56:56.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.541455  6166 consensus_queue.cc:237] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:56.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:56.542384  4838 catalog_manager.cc:5495] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc reported cstate change: term changed from 5 to 6, leader changed from 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66) to 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65). New cstate: current_term: 6 leader_uuid: "12bc83bf6ff640bc9820d875747f49cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } health_report { overall_health: UNKNOWN } } }
07:56:56.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:56.967147  5850 consensus_peers.cc:487] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc -> Peer 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353): Couldn't send request to peer 137d6728589143c1a8ae1e4cc280df77. Status: Network error: Client connection negotiation failed: client connection to 127.4.157.66:34353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:57.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.134372  6092 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 6 FOLLOWER]: Refusing update from remote peer 12bc83bf6ff640bc9820d875747f49cc: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
07:56:57.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.134716  6166 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
07:56:57.172 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:57.172 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:57.172 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1983), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [3ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [24ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [25ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [25ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [41ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [46ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [46ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [81ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [104ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [106ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [106ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [220ms] refreshing cache from master, [220ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [222ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [223ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [226ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [226ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [699ms] refreshing cache from master, [699ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [701ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [703ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [705ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [705ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [1980ms] refreshing cache from master, [1980ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [1982ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@1995529524(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@1c512b67 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@731484255 after retry RPC@2003494086) -> (continuation of Deferred@1319156764 after retry RPC@1858874842) -> (continuation of Deferred@1772116630 after retry RPC@172866412) -> (continuation of Deferred@78892860 after retry RPC@1665307165) -> (continuation of Deferred@880966255 after retry RPC@1481179659), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b3fa81a -> wakeup thread Time-limited test -> (continuation of Deferred@731484255 after retry RPC@2003494086) -> (continuation of Deferred@1319156764 after retry RPC@1858874842) -> (continuation of Deferred@1772116630 after retry RPC@172866412) -> (continuation of Deferred@78892860 after retry RPC@1665307165) -> (continuation of Deferred@880966255 after retry RPC@1481179659))) after lookup
07:56:57.173 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:57.175 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] Successfully connected to peer
07:56:57.194 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x635ac237, L:/127.0.0.1:53818 - R:/127.4.157.65:34783] using TOKEN/null
07:56:57.206 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.157.66:34353
07:56:57.207 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.208132  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/logs
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.66:34353
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.66
07:56:57.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:57.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36277
07:56:57.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.66
07:56:57.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:57.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:57.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:57.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:57.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.297155  6194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:57.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.297394  6194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:57.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.297427  6194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:57.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.298856  6194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:57.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.298949  6194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.66
07:56:57.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.300427  6194 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:57.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:57.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:57.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:57.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:57.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data
07:56:57.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:57.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.66:34353
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.66
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36277
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.6194
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.66
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/logs
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:57.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:57.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:57.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.300704  6194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:57.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.301133  6194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:57.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.307091  6200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:57.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.307080  6199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:57.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:57.307075  6202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:57.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.307300  6194 server_base.cc:1034] running on GCE node
07:56:57.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.307723  6194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:57.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.307986  6194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:57.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.309136  6194 hybrid_clock.cc:648] HybridClock initialized: now 1783497417309110 us; error 55 us; skew 500 ppm
07:56:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.310529  6194 webserver.cc:460] Webserver started at http://127.4.157.66:36277/ using document root <none> and password file <none>
07:56:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.310863  6194 fs_manager.cc:359] Metadata directory not provided
07:56:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.310943  6194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.312275  6194 fs_manager.cc:709] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
07:56:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.313124  6208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.313362  6194 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
07:56:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.313446  6194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "137d6728589143c1a8ae1e4cc280df77"
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.313836  6194 fs_report.cc:383] FS layout report
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/data
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:57.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:57.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:57.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:57.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.355661  6194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:57.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.356045  6194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:57.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.356232  6194 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:57.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.356523  6194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:56:57.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.357103  6215 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
07:56:57.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.358595  6194 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:57.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.358661  6194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s	user 0.000s	sys 0.000s
07:56:57.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.358696  6194 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
07:56:57.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.359357  6194 ts_tablet_manager.cc:610] Registered 1 tablets
07:56:57.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.359419  6194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
07:56:57.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.359453  6215 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Bootstrap starting.
07:56:57.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.366200  6194 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.66:34353
07:56:57.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.366480  6322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.66:34353 every 8 connection(s)
07:56:57.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.366681  6194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:56:57.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.369915  6215 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Log is configured to *not* fsync() on all Append() calls
07:56:57.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.373454  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 6194
07:56:57.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.373761  6215 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:57.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.374060  6215 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Bootstrap complete.
07:56:57.374 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:57.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.374958  6215 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.006s
07:56:57.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.375870  6215 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:57.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.376113  6215 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137d6728589143c1a8ae1e4cc280df77, State: Initialized, Role: FOLLOWER
07:56:57.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.376248  6215 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:57.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.376561  6215 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
07:56:57.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.376771  6325 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:56:57.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.376865  6325 heartbeater.cc:464] Registering TS with master...
07:56:57.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.377060  6325 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:56:57.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.377071  6323 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:56:57.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.377140  6323 heartbeater.cc:464] Registering TS with master...
07:56:57.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.377207  6323 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:56:57.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.377681  4839 ts_manager.cc:194] Re-registered known tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:56:57.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.377875  4911 ts_manager.cc:194] Re-registered known tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:56:57.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.378180  4839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.66:35695
07:56:57.378 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:97)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
07:56:57.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.378903  6325 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:56:57.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.379266  6324 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:56:57.379 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:79)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
07:56:57.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.379328  6324 heartbeater.cc:464] Registering TS with master...
07:56:57.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.379371  6324 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:56:57.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.379660  4767 ts_manager.cc:194] Re-registered known tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:56:57.383 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:57.383 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.157.65:34783
07:56:57.384 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:56:57.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:57.384260  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 5832
07:56:57.389 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] handling channelInactive
07:56:57.389 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:57.389 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state READY due to: connection closed
07:56:57.390 [DEBUG - Time-limited test] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:57.391 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.391 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.391 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.411 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:57.411 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:57.412 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=22), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@8960049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@d037329 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@674654a9 -> wakeup thread Time-limited test)) after lookup
07:56:57.412 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:57.415 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.415 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.415 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.430 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:57.431 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:57.431 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [23ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [26ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [26ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [41ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@8960049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@d037329 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2043033205 after retry RPC@745592568), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@674654a9 -> wakeup thread Time-limited test -> (continuation of Deferred@2043033205 after retry RPC@745592568))) after lookup
07:56:57.432 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:57.433 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.433 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.433 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.471 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:57.471 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:57.472 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=82), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [23ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [26ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [26ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [41ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [44ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [44ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [81ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@8960049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@d037329 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2043033205 after retry RPC@745592568) -> (continuation of Deferred@910444253 after retry RPC@2085313122), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@674654a9 -> wakeup thread Time-limited test -> (continuation of Deferred@2043033205 after retry RPC@745592568) -> (continuation of Deferred@910444253 after retry RPC@2085313122))) after lookup
07:56:57.473 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:57.474 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.475 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.475 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.551 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:57.551 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:57.551 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=162), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [23ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [26ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [26ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [41ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [44ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [44ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [81ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [84ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [86ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [86ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [159ms] refreshing cache from master, [160ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [161ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@8960049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@d037329 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2043033205 after retry RPC@745592568) -> (continuation of Deferred@910444253 after retry RPC@2085313122) -> (continuation of Deferred@743015851 after retry RPC@183976458), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@674654a9 -> wakeup thread Time-limited test -> (continuation of Deferred@2043033205 after retry RPC@745592568) -> (continuation of Deferred@910444253 after retry RPC@2085313122) -> (continuation of Deferred@743015851 after retry RPC@183976458))) after lookup
07:56:57.552 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:57.553 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.553 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.556 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.991 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:57.991 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:56:57.991 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=602), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [23ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [26ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [26ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [41ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [44ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [44ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [81ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [84ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [86ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [86ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [159ms] refreshing cache from master, [160ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [161ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [163ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [167ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [168ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [599ms] refreshing cache from master, [600ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [601ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@8960049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@d037329 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2043033205 after retry RPC@745592568) -> (continuation of Deferred@910444253 after retry RPC@2085313122) -> (continuation of Deferred@743015851 after retry RPC@183976458) -> (continuation of Deferred@1545865352 after retry RPC@1303002892), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@674654a9 -> wakeup thread Time-limited test -> (continuation of Deferred@2043033205 after retry RPC@745592568) -> (continuation of Deferred@910444253 after retry RPC@2085313122) -> (continuation of Deferred@743015851 after retry RPC@183976458) -> (continuation of Deferred@1545865352 after retry RPC@1303002892))) after lookup
07:56:57.992 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:56:57.994 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.994 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:57.994 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:56:58.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.706807  6336 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 12bc83bf6ff640bc9820d875747f49cc)
07:56:58.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.706897  6336 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:58.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.707231  6336 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:58.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.707765  6037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111)
07:56:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.708276  6037 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783): Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111)
07:56:58.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.711381  6277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "f772946b4e6940dda07baac926a093bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "137d6728589143c1a8ae1e4cc280df77" is_pre_election: true
07:56:58.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.711520  6277 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f772946b4e6940dda07baac926a093bf in term 5.
07:56:58.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.711825  6040 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 137d6728589143c1a8ae1e4cc280df77, f772946b4e6940dda07baac926a093bf; no voters: 12bc83bf6ff640bc9820d875747f49cc
07:56:58.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.711951  6336 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 6 FOLLOWER]: Leader pre-election won for term 7
07:56:58.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.711997  6336 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 6 FOLLOWER]: Starting leader election (detected failure of leader 12bc83bf6ff640bc9820d875747f49cc)
07:56:58.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.712021  6336 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 6 FOLLOWER]: Advancing to term 7
07:56:58.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.712628  6336 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:58.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.712793  6336 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 7 election: Requested vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:58.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.712984  6277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "f772946b4e6940dda07baac926a093bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "137d6728589143c1a8ae1e4cc280df77"
07:56:58.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.713089  6277 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 5 FOLLOWER]: Advancing to term 7
07:56:58.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:58.713518  6037 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783): Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111)
07:56:58.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.714394  6277 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f772946b4e6940dda07baac926a093bf in term 7.
07:56:58.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.714622  6040 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 137d6728589143c1a8ae1e4cc280df77, f772946b4e6940dda07baac926a093bf; no voters: 12bc83bf6ff640bc9820d875747f49cc
07:56:58.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.714749  6336 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 7 FOLLOWER]: Leader election won for term 7
07:56:58.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.714922  6336 raft_consensus.cc:686] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 7 LEADER]: Becoming Leader. State: Replica: f772946b4e6940dda07baac926a093bf, State: Running, Role: LEADER
07:56:58.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.715176  6336 consensus_queue.cc:237] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:58.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:58.716064  4839 catalog_manager.cc:5495] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf reported cstate change: term changed from 6 to 7, leader changed from 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65) to f772946b4e6940dda07baac926a093bf (127.4.157.67). New cstate: current_term: 7 leader_uuid: "f772946b4e6940dda07baac926a093bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } health_report { overall_health: HEALTHY } } }
07:56:59.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.109896  6037 consensus_peers.cc:487] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf -> Peer 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783): Couldn't send request to peer 12bc83bf6ff640bc9820d875747f49cc. Status: Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:56:59.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.149513  6277 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 7 FOLLOWER]: Refusing update from remote peer f772946b4e6940dda07baac926a093bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
07:56:59.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.149941  6336 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
07:56:59.811 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:59.811 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:59.812 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2422), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [3ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [21ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [23ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [26ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [26ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [41ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [43ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [44ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [44ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [81ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [84ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [86ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [86ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [159ms] refreshing cache from master, [160ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [161ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [163ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [167ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [168ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [599ms] refreshing cache from master, [600ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [601ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [603ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [605ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [606ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [2420ms] refreshing cache from master, [2420ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [2422ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@8960049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@d037329 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2043033205 after retry RPC@745592568) -> (continuation of Deferred@910444253 after retry RPC@2085313122) -> (continuation of Deferred@743015851 after retry RPC@183976458) -> (continuation of Deferred@1545865352 after retry RPC@1303002892) -> (continuation of Deferred@1050830706 after retry RPC@585924085), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@674654a9 -> wakeup thread Time-limited test -> (continuation of Deferred@2043033205 after retry RPC@745592568) -> (continuation of Deferred@910444253 after retry RPC@2085313122) -> (continuation of Deferred@743015851 after retry RPC@183976458) -> (continuation of Deferred@1545865352 after retry RPC@1303002892) -> (continuation of Deferred@1050830706 after retry RPC@585924085))) after lookup
07:56:59.812 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:59.813 [DEBUG - kudu-nio-0] (Connection.java:757) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] Successfully connected to peer
07:56:59.828 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x4291d6c1, L:/127.0.0.1:38866 - R:/127.4.157.67:39629] using TOKEN/null
07:56:59.839 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.157.65:34783
07:56:59.839 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.840232  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data
07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/logs
07:56:59.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.65:34783
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.65
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40963
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.65
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:56:59.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.923151  6367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:56:59.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.923467  6367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:56:59.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.923499  6367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:56:59.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.924713  6367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:56:59.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.924767  6367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.65
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.926098  6367 tablet_server_runner.cc:78] Tablet server non-default flags:
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.65:34783
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.65
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40963
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.6367
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.65
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/logs
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.926311  6367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:56:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.926560  6367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:56:59.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.932085  6373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:59.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.932087  6375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:59.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:56:59.932265  6372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:56:59.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.932318  6367 server_base.cc:1034] running on GCE node
07:56:59.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.932511  6367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:56:59.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.932765  6367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:56:59.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.933903  6367 hybrid_clock.cc:648] HybridClock initialized: now 1783497419933885 us; error 42 us; skew 500 ppm
07:56:59.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.935092  6367 webserver.cc:460] Webserver started at http://127.4.157.65:40963/ using document root <none> and password file <none>
07:56:59.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.935278  6367 fs_manager.cc:359] Metadata directory not provided
07:56:59.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.935321  6367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:56:59.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.936594  6367 fs_manager.cc:709] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
07:56:59.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.937415  6381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:56:59.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.937606  6367 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.937677  6367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "12bc83bf6ff640bc9820d875747f49cc"
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.938024  6367 fs_report.cc:383] FS layout report
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/data
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:56:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:56:59.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.953433  6367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:56:59.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.953727  6367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:56:59.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.953833  6367 kserver.cc:163] Server-wide thread pool size limit: 3276
07:56:59.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.954010  6367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:56:59.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.954528  6388 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
07:56:59.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.955976  6367 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:56:59.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.956048  6367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s	user 0.000s	sys 0.000s
07:56:59.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.956084  6367 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
07:56:59.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.956817  6367 ts_tablet_manager.cc:610] Registered 1 tablets
07:56:59.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.956878  6367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
07:56:59.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.956909  6388 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Bootstrap starting.
07:56:59.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.964627  6367 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.65:34783
07:56:59.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.964828  6495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.65:34783 every 8 connection(s)
07:56:59.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.964964  6388 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Log is configured to *not* fsync() on all Append() calls
07:56:59.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.965157  6367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:56:59.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.965366  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 6367
07:56:59.966 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:59.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.969559  6388 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:56:59.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.969890  6388 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Bootstrap complete.
07:56:59.970 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:56:59.971 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.157.67:39629
07:56:59.971 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:56:59.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.971056  6388 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.004s
07:56:59.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.971853  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 6020
07:56:59.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.973390  6388 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:59.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.973696  6388 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bc83bf6ff640bc9820d875747f49cc, State: Initialized, Role: FOLLOWER
07:56:59.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.973856  6388 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:56:59.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.974099  6388 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
07:56:59.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.977047  6498 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:56:59.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.977103  6499 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:56:59.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.977151  6498 heartbeater.cc:464] Registering TS with master...
07:56:59.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.977154  6499 heartbeater.cc:464] Registering TS with master...
07:56:59.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.977320  6499 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:56:59.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.977322  6498 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:56:59.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.977948  4839 ts_manager.cc:194] Re-registered known tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:59.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.978024  4767 ts_manager.cc:194] Re-registered known tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:59.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.978443  4839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.65:50607
07:56:59.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.979347  6499 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:56:59.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.980268  6497 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:56:59.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.980341  6497 heartbeater.cc:464] Registering TS with master...
07:56:59.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.980391  6497 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:56:59.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:56:59.980770  4911 ts_manager.cc:194] Re-registered known tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:56:59.981 [DEBUG - kudu-nio-0] (Connection.java:239) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] handling channelInactive
07:56:59.981 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state READY due to: connection closed
07:56:59.983 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:56:59.983 [DEBUG - Time-limited test] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:59.985 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:59.985 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:59.985 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:59.991 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:56:59.991 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:56:59.992 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=8), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [3ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [7ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@756581031(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5b4f73ef -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c0d4e5 -> wakeup thread Time-limited test)) after lookup
07:56:59.992 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:56:59.993 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:59.993 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:56:59.994 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.011 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:00.012 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:57:00.012 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [3ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [7ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [9ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [10ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [11ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@756581031(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5b4f73ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c0d4e5 -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567))) after lookup
07:57:00.013 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:00.015 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.015 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.015 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.031 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:00.031 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:57:00.032 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=48), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [3ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [7ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [9ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [10ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [11ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [29ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [32ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [32ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [48ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@756581031(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5b4f73ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567) -> (continuation of Deferred@789149746 after retry RPC@35001591), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c0d4e5 -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567) -> (continuation of Deferred@789149746 after retry RPC@35001591))) after lookup
07:57:00.032 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:00.033 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.033 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.033 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.171 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:00.172 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:57:00.172 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=189), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [3ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [7ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [9ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [10ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [11ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [29ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [32ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [32ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [48ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [49ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [50ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [50ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [188ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@756581031(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5b4f73ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567) -> (continuation of Deferred@789149746 after retry RPC@35001591) -> (continuation of Deferred@1726071603 after retry RPC@570506098), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c0d4e5 -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567) -> (continuation of Deferred@789149746 after retry RPC@35001591) -> (continuation of Deferred@1726071603 after retry RPC@570506098))) after lookup
07:57:00.173 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:00.175 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.175 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.175 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.271 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:00.271 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:57:00.271 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=288), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [3ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [7ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [9ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [10ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [11ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [29ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [32ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [32ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [48ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [49ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [50ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [50ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [188ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [190ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [192ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [192ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [285ms] refreshing cache from master, [285ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [287ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@756581031(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5b4f73ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567) -> (continuation of Deferred@789149746 after retry RPC@35001591) -> (continuation of Deferred@1726071603 after retry RPC@570506098) -> (continuation of Deferred@1617413710 after retry RPC@1259776923), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c0d4e5 -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567) -> (continuation of Deferred@789149746 after retry RPC@35001591) -> (continuation of Deferred@1726071603 after retry RPC@570506098) -> (continuation of Deferred@1617413710 after retry RPC@1259776923))) after lookup
07:57:00.272 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:00.273 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.274 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:00.274 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:01.031 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:01.031 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:57:01.032 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1048), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [3ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [7ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [9ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [10ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [11ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [29ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [32ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [32ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [48ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [49ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [50ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [50ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [188ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [190ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [192ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [192ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [285ms] refreshing cache from master, [285ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [287ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [289ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [291ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [291ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [1045ms] refreshing cache from master, [1045ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [1047ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@756581031(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5b4f73ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567) -> (continuation of Deferred@789149746 after retry RPC@35001591) -> (continuation of Deferred@1726071603 after retry RPC@570506098) -> (continuation of Deferred@1617413710 after retry RPC@1259776923) -> (continuation of Deferred@426362923 after retry RPC@1512543231), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c0d4e5 -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567) -> (continuation of Deferred@789149746 after retry RPC@35001591) -> (continuation of Deferred@1726071603 after retry RPC@570506098) -> (continuation of Deferred@1617413710 after retry RPC@1259776923) -> (continuation of Deferred@426362923 after retry RPC@1512543231))) after lookup
07:57:01.033 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:01.035 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:01.035 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:01.035 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:01.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.339051  6509 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f772946b4e6940dda07baac926a093bf)
07:57:01.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.339164  6509 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:01.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.339490  6509 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:01.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:01.340026  6212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:57:01.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:01.340499  6212 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:57:01.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.343556  6450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "137d6728589143c1a8ae1e4cc280df77" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "12bc83bf6ff640bc9820d875747f49cc" is_pre_election: true
07:57:01.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.343743  6450 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137d6728589143c1a8ae1e4cc280df77 in term 6.
07:57:01.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.344013  6209 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, 137d6728589143c1a8ae1e4cc280df77; no voters: f772946b4e6940dda07baac926a093bf
07:57:01.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.344177  6509 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 7 FOLLOWER]: Leader pre-election won for term 8
07:57:01.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.344242  6509 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f772946b4e6940dda07baac926a093bf)
07:57:01.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.344259  6509 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 7 FOLLOWER]: Advancing to term 8
07:57:01.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.344846  6509 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:01.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.345006  6509 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 8 election: Requested vote from peers 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:01.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.345176  6450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "137d6728589143c1a8ae1e4cc280df77" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "12bc83bf6ff640bc9820d875747f49cc"
07:57:01.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.345326  6450 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 6 FOLLOWER]: Advancing to term 8
07:57:01.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:01.345547  6212 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:57:01.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.346534  6450 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137d6728589143c1a8ae1e4cc280df77 in term 8.
07:57:01.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.346757  6209 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, 137d6728589143c1a8ae1e4cc280df77; no voters: f772946b4e6940dda07baac926a093bf
07:57:01.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.346901  6509 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 8 FOLLOWER]: Leader election won for term 8
07:57:01.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.347045  6509 raft_consensus.cc:686] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 8 LEADER]: Becoming Leader. State: Replica: 137d6728589143c1a8ae1e4cc280df77, State: Running, Role: LEADER
07:57:01.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.347177  6509 consensus_queue.cc:237] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:01.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.348040  4839 catalog_manager.cc:5495] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 reported cstate change: term changed from 7 to 8, leader changed from f772946b4e6940dda07baac926a093bf (127.4.157.67) to 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66). New cstate: current_term: 8 leader_uuid: "137d6728589143c1a8ae1e4cc280df77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } health_report { overall_health: UNKNOWN } } }
07:57:01.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:01.803308  6212 consensus_peers.cc:487] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 -> Peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Couldn't send request to peer f772946b4e6940dda07baac926a093bf. Status: Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:57:01.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.856581  6450 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 8 FOLLOWER]: Refusing update from remote peer 137d6728589143c1a8ae1e4cc280df77: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
07:57:01.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.857064  6509 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
07:57:01.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:01.859368  6528 mvcc.cc:205] Tried to move back new op lower bound from 7305205431654907904 to 7305205427057897472. Current Snapshot: MvccSnapshot[applied={T|T < 7305205431623446528}]
07:57:03.170 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:03.171 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:03.171 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=3188), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [2ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [3ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [7ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [9ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [10ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [11ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [29ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [32ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [32ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [48ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [49ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [50ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [50ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [188ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [190ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [192ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [192ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [285ms] refreshing cache from master, [285ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [287ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [289ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [291ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [291ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [1045ms] refreshing cache from master, [1045ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [1047ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [1050ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [1052ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [1052ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [3185ms] refreshing cache from master, [3185ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [3187ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@756581031(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@5b4f73ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567) -> (continuation of Deferred@789149746 after retry RPC@35001591) -> (continuation of Deferred@1726071603 after retry RPC@570506098) -> (continuation of Deferred@1617413710 after retry RPC@1259776923) -> (continuation of Deferred@426362923 after retry RPC@1512543231) -> (continuation of Deferred@1388327567 after retry RPC@874445635), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c0d4e5 -> wakeup thread Time-limited test -> (continuation of Deferred@573557362 after retry RPC@1613341567) -> (continuation of Deferred@789149746 after retry RPC@35001591) -> (continuation of Deferred@1726071603 after retry RPC@570506098) -> (continuation of Deferred@1617413710 after retry RPC@1259776923) -> (continuation of Deferred@426362923 after retry RPC@1512543231) -> (continuation of Deferred@1388327567 after retry RPC@874445635))) after lookup
07:57:03.172 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:57:03.173 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] Successfully connected to peer
07:57:03.195 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x02945c3a, L:/127.0.0.1:42400 - R:/127.4.157.66:34353] using TOKEN/null
07:57:03.211 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.157.67:39629
07:57:03.211 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.212122  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/logs
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:57:03.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:57:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:57:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:57:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:57:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.67:39629
07:57:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.67
07:57:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:57:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46573
07:57:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.67
07:57:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:57:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:57:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:57:03.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:57:03.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:03.304721  6550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:57:03.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:03.304932  6550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:57:03.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:03.304955  6550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:57:03.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:03.306213  6550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:57:03.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:03.306269  6550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.67
07:57:03.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.307745  6550 tablet_server_runner.cc:78] Tablet server non-default flags:
07:57:03.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:57:03.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:57:03.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:57:03.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:57:03.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.67:39629
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.67
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46573
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.6550
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.67
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/logs
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.307950  6550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:57:03.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.308265  6550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:57:03.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:03.313894  6555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:03.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:03.313944  6556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:03.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.313951  6550 server_base.cc:1034] running on GCE node
07:57:03.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:03.313922  6558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:03.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.314589  6550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:57:03.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.314805  6550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:57:03.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.315937  6550 hybrid_clock.cc:648] HybridClock initialized: now 1783497423315903 us; error 55 us; skew 500 ppm
07:57:03.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.317390  6550 webserver.cc:460] Webserver started at http://127.4.157.67:46573/ using document root <none> and password file <none>
07:57:03.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.317708  6550 fs_manager.cc:359] Metadata directory not provided
07:57:03.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.317790  6550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:57:03.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.319203  6550 fs_manager.cc:709] Time spent opening directory manager: real 0.001s	user 0.000s	sys 0.002s
07:57:03.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.319962  6564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:57:03.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.320163  6550 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
07:57:03.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.320288  6550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:57:03.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f772946b4e6940dda07baac926a093bf"
07:57:03.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.320801  6550 fs_report.cc:383] FS layout report
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/data
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:57:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:57:03.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.342305  6550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:57:03.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.342713  6550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:57:03.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.342890  6550 kserver.cc:163] Server-wide thread pool size limit: 3276
07:57:03.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.343201  6550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:57:03.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.343850  6571 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
07:57:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.345247  6550 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:57:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.345314  6550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s	user 0.000s	sys 0.000s
07:57:03.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.345363  6550 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
07:57:03.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.346063  6550 ts_tablet_manager.cc:610] Registered 1 tablets
07:57:03.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.346125  6550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
07:57:03.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.346167  6571 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap starting.
07:57:03.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.353937  6550 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.67:39629
07:57:03.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.354041  6678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.67:39629 every 8 connection(s)
07:57:03.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.354422  6550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:57:03.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.356003  6571 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Log is configured to *not* fsync() on all Append() calls
07:57:03.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.358148  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 6550
07:57:03.358 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:57:03.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.361059  6571 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:57:03.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.361420  6571 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap complete.
07:57:03.362 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:57:03.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.362617  6571 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.008s
07:57:03.363 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.157.66:34353
07:57:03.364 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:57:03.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.364169  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 6194
07:57:03.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.365190  6681 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:57:03.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.365288  6681 heartbeater.cc:464] Registering TS with master...
07:57:03.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.365631  6681 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:57:03.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.366045  6680 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:57:03.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.366101  6680 heartbeater.cc:464] Registering TS with master...
07:57:03.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.366163  6680 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:57:03.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.364136  6571 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:03.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.366459  6571 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: f772946b4e6940dda07baac926a093bf, State: Initialized, Role: FOLLOWER
07:57:03.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.366613  6571 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:03.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.366907  6571 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.002s
07:57:03.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.367444  4838 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:03.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.367877  4838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.67:36745
07:57:03.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.368198  4767 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:03.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.368599  6679 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:57:03.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.368674  6679 heartbeater.cc:464] Registering TS with master...
07:57:03.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.368911  6679 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:57:03.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.369081  6681 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:57:03.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:03.369407  4910 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:03.371 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:57:03.371 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] handling channelInactive
07:57:03.371 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state READY due to: connection closed
07:57:03.371 [DEBUG - Time-limited test] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:57:03.373 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.373 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.373 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.390 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:03.391 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:03.391 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [2ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [19ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@313097735(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@24eec52a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60507edd -> wakeup thread Time-limited test)) after lookup
07:57:03.392 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:57:03.393 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.393 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.393 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.411 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:03.412 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:03.412 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [2ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [19ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [20ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [22ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [22ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@313097735(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@24eec52a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@690230581 after retry RPC@1138331390), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60507edd -> wakeup thread Time-limited test -> (continuation of Deferred@690230581 after retry RPC@1138331390))) after lookup
07:57:03.413 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:57:03.415 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.415 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.415 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.451 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:03.451 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:03.451 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [2ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [19ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [20ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [22ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [22ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [41ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [44ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [44ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [79ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@313097735(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@24eec52a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@690230581 after retry RPC@1138331390) -> (continuation of Deferred@11119819 after retry RPC@260602974), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60507edd -> wakeup thread Time-limited test -> (continuation of Deferred@690230581 after retry RPC@1138331390) -> (continuation of Deferred@11119819 after retry RPC@260602974))) after lookup
07:57:03.452 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:57:03.453 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.453 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.453 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.571 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:03.571 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:03.572 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=200), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [2ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [19ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [20ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [22ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [22ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [41ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [44ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [44ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [79ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [80ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [82ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [82ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [198ms] refreshing cache from master, [198ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [199ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@313097735(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@24eec52a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@690230581 after retry RPC@1138331390) -> (continuation of Deferred@11119819 after retry RPC@260602974) -> (continuation of Deferred@1441916530 after retry RPC@2127989632), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60507edd -> wakeup thread Time-limited test -> (continuation of Deferred@690230581 after retry RPC@1138331390) -> (continuation of Deferred@11119819 after retry RPC@260602974) -> (continuation of Deferred@1441916530 after retry RPC@2127989632))) after lookup
07:57:03.572 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:57:03.574 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.574 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.574 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.792 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:03.792 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:03.792 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=421), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [2ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [19ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [20ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [22ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [22ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [41ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [44ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [44ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [79ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [80ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [82ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [82ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [198ms] refreshing cache from master, [198ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [199ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [201ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [203ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [203ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [418ms] refreshing cache from master, [418ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [420ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@313097735(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@24eec52a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@690230581 after retry RPC@1138331390) -> (continuation of Deferred@11119819 after retry RPC@260602974) -> (continuation of Deferred@1441916530 after retry RPC@2127989632) -> (continuation of Deferred@1600717553 after retry RPC@355010583), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60507edd -> wakeup thread Time-limited test -> (continuation of Deferred@690230581 after retry RPC@1138331390) -> (continuation of Deferred@11119819 after retry RPC@260602974) -> (continuation of Deferred@1441916530 after retry RPC@2127989632) -> (continuation of Deferred@1600717553 after retry RPC@355010583))) after lookup
07:57:03.793 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:57:03.794 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.794 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state CONNECTING due to: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:03.795 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353
07:57:04.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.711819  6692 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 137d6728589143c1a8ae1e4cc280df77)
07:57:04.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.711910  6692 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:04.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.712188  6692 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:04.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:04.712747  6385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.66:34353: connect: Connection refused (error 111)
07:57:04.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:04.713362  6385 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353): Network error: Client connection negotiation failed: client connection to 127.4.157.66:34353: connect: Connection refused (error 111)
07:57:04.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.716238  6633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "12bc83bf6ff640bc9820d875747f49cc" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "f772946b4e6940dda07baac926a093bf" is_pre_election: true
07:57:04.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.716378  6633 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bc83bf6ff640bc9820d875747f49cc in term 7.
07:57:04.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.716610  6385 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, f772946b4e6940dda07baac926a093bf; no voters: 137d6728589143c1a8ae1e4cc280df77
07:57:04.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.716744  6692 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 8 FOLLOWER]: Leader pre-election won for term 9
07:57:04.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.716809  6692 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 8 FOLLOWER]: Starting leader election (detected failure of leader 137d6728589143c1a8ae1e4cc280df77)
07:57:04.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.716826  6692 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 8 FOLLOWER]: Advancing to term 9
07:57:04.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.717674  6692 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:04.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.717906  6692 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 9 election: Requested vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:04.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.718182  6633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "12bc83bf6ff640bc9820d875747f49cc" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "f772946b4e6940dda07baac926a093bf"
07:57:04.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.718303  6633 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 7 FOLLOWER]: Advancing to term 9
07:57:04.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:04.718555  6385 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353): Network error: Client connection negotiation failed: client connection to 127.4.157.66:34353: connect: Connection refused (error 111)
07:57:04.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.719600  6633 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bc83bf6ff640bc9820d875747f49cc in term 9.
07:57:04.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.719774  6385 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, f772946b4e6940dda07baac926a093bf; no voters: 137d6728589143c1a8ae1e4cc280df77
07:57:04.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.719870  6692 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 9 FOLLOWER]: Leader election won for term 9
07:57:04.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.720000  6692 raft_consensus.cc:686] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 9 LEADER]: Becoming Leader. State: Replica: 12bc83bf6ff640bc9820d875747f49cc, State: Running, Role: LEADER
07:57:04.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.720108  6692 consensus_queue.cc:237] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:04.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:04.720777  4838 catalog_manager.cc:5495] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc reported cstate change: term changed from 8 to 9, leader changed from 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66) to 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65). New cstate: current_term: 9 leader_uuid: "12bc83bf6ff640bc9820d875747f49cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } health_report { overall_health: UNKNOWN } } }
07:57:05.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:05.110059  6385 consensus_peers.cc:487] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc -> Peer 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353): Couldn't send request to peer 137d6728589143c1a8ae1e4cc280df77. Status: Network error: Client connection negotiation failed: client connection to 127.4.157.66:34353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:57:05.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.202881  6633 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 9 FOLLOWER]: Refusing update from remote peer 12bc83bf6ff640bc9820d875747f49cc: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
07:57:05.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.203317  6692 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
07:57:05.791 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:05.791 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:05.792 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2420), Traces: [0ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [2ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [2ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [19ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [20ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [22ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [22ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [41ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [44ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [44ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [79ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [80ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [82ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [82ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [198ms] refreshing cache from master, [198ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [199ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [201ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [203ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [203ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [418ms] refreshing cache from master, [418ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [420ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [422ms] sending RPC to server 137d6728589143c1a8ae1e4cc280df77, [424ms] received response from server 137d6728589143c1a8ae1e4cc280df77: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [424ms] delaying RPC due to: Network error: Failed to connect to peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353): Connection refused: /127.4.157.66:34353, [2417ms] refreshing cache from master, [2417ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [2419ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@313097735(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@24eec52a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@690230581 after retry RPC@1138331390) -> (continuation of Deferred@11119819 after retry RPC@260602974) -> (continuation of Deferred@1441916530 after retry RPC@2127989632) -> (continuation of Deferred@1600717553 after retry RPC@355010583) -> (continuation of Deferred@845913795 after retry RPC@880584112), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60507edd -> wakeup thread Time-limited test -> (continuation of Deferred@690230581 after retry RPC@1138331390) -> (continuation of Deferred@11119819 after retry RPC@260602974) -> (continuation of Deferred@1441916530 after retry RPC@2127989632) -> (continuation of Deferred@1600717553 after retry RPC@355010583) -> (continuation of Deferred@845913795 after retry RPC@880584112))) after lookup
07:57:05.794 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:57:05.796 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] Successfully connected to peer
07:57:05.811 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x317df8f5, L:/127.0.0.1:49730 - R:/127.4.157.65:34783] using TOKEN/null
07:57:05.820 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.157.66:34353
07:57:05.820 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:57:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.820616  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:57:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:57:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:57:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data
07:57:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:57:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:57:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:57:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:57:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:57:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/logs
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.66:34353
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.66
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36277
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.66
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:57:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:57:05.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:05.902086  6722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:57:05.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:05.902329  6722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:57:05.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:05.902361  6722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:57:05.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:05.903659  6722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:57:05.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:05.903743  6722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.66
07:57:05.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.905166  6722 tablet_server_runner.cc:78] Tablet server non-default flags:
07:57:05.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:57:05.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:57:05.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:57:05.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.66:34353
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.66
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36277
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.6722
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.66
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/logs
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:57:05.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:57:05.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:57:05.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.905419  6722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:57:05.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.905771  6722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:57:05.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:05.911175  6727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:05.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:05.911197  6728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:05.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:05.911226  6730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:05.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.911712  6722 server_base.cc:1034] running on GCE node
07:57:05.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.911976  6722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:57:05.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.912204  6722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:57:05.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.913391  6722 hybrid_clock.cc:648] HybridClock initialized: now 1783497425913373 us; error 34 us; skew 500 ppm
07:57:05.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.919878  6722 webserver.cc:460] Webserver started at http://127.4.157.66:36277/ using document root <none> and password file <none>
07:57:05.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.920176  6722 fs_manager.cc:359] Metadata directory not provided
07:57:05.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.920259  6722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:57:05.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.921833  6722 fs_manager.cc:709] Time spent opening directory manager: real 0.001s	user 0.002s	sys 0.000s
07:57:05.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.922679  6736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:57:05.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.922878  6722 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
07:57:05.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.922991  6722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:57:05.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "137d6728589143c1a8ae1e4cc280df77"
07:57:05.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:57:05.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.923439  6722 fs_report.cc:383] FS layout report
07:57:05.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:57:05.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:57:05.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/wal
07:57:05.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/data
07:57:05.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:57:05.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:57:05.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:57:05.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:57:05.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:57:05.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:57:05.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:57:05.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:57:05.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:57:05.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:57:05.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:57:05.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.935611  6722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:57:05.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.936007  6722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:57:05.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.936182  6722 kserver.cc:163] Server-wide thread pool size limit: 3276
07:57:05.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.936512  6722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:57:05.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.937088  6743 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
07:57:05.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.938458  6722 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:57:05.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.938563  6722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s	user 0.000s	sys 0.000s
07:57:05.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.938628  6722 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
07:57:05.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.939256  6722 ts_tablet_manager.cc:610] Registered 1 tablets
07:57:05.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.939289  6722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
07:57:05.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.939421  6743 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Bootstrap starting.
07:57:05.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.946412  6722 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.66:34353
07:57:05.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.946496  6850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.66:34353 every 8 connection(s)
07:57:05.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.946940  6722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-1/data/info.pb
07:57:05.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.949750  6743 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Log is configured to *not* fsync() on all Append() calls
07:57:05.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.954511  6853 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:57:05.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.954627  6853 heartbeater.cc:464] Registering TS with master...
07:57:05.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.954874  6853 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:57:05.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.955021  6851 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:57:05.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.955067  6851 heartbeater.cc:464] Registering TS with master...
07:57:05.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.955175  6851 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:57:05.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.954619  6743 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:57:05.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.955358  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 6722
07:57:05.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.955435  4839 ts_manager.cc:194] Re-registered known tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:57:05.955 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:57:05.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.955647  6743 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Bootstrap complete.
07:57:05.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.955919  4839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.66:48611
07:57:05.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.957155  4910 ts_manager.cc:194] Re-registered known tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:57:05.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.957558  6743 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.008s
07:57:05.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.957970  6852 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:57:05.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.958029  6852 heartbeater.cc:464] Registering TS with master...
07:57:05.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.958102  6852 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:57:05.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.958890  6743 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:05.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.959211  6743 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137d6728589143c1a8ae1e4cc280df77, State: Initialized, Role: FOLLOWER
07:57:05.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.959342  6743 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:05.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.959599  6853 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:57:05.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.959746  6743 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
07:57:05.960 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:57:05.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.960239  4767 ts_manager.cc:194] Re-registered known tserver with Master: 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353)
07:57:05.960 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.157.65:34783
07:57:05.961 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:57:05.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:05.961293  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 6367
07:57:05.967 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:57:05.967 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] handling channelInactive
07:57:05.967 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state READY due to: connection closed
07:57:05.967 [DEBUG - Time-limited test] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:57:05.969 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:05.969 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:05.969 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:05.991 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:05.991 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:05.992 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [2ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [23ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@724461488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@549ed413 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d3ef481 -> wakeup thread Time-limited test)) after lookup
07:57:05.992 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:57:05.995 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:05.995 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:05.995 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.011 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:06.011 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:06.011 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [2ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [23ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [25ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [28ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [28ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [43ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@724461488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@549ed413 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2090739197 after retry RPC@1445878803), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d3ef481 -> wakeup thread Time-limited test -> (continuation of Deferred@2090739197 after retry RPC@1445878803))) after lookup
07:57:06.012 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:57:06.013 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.013 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.013 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.050 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:06.050 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:06.051 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [2ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [23ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [25ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [28ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [28ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [43ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [45ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [46ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [46ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [83ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@724461488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@549ed413 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2090739197 after retry RPC@1445878803) -> (continuation of Deferred@968441553 after retry RPC@1575500337), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d3ef481 -> wakeup thread Time-limited test -> (continuation of Deferred@2090739197 after retry RPC@1445878803) -> (continuation of Deferred@968441553 after retry RPC@1575500337))) after lookup
07:57:06.052 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:57:06.054 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.054 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.054 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.111 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:06.111 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:06.111 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=144), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [2ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [23ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [25ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [28ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [28ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [43ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [45ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [46ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [46ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [83ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [84ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [87ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [87ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [141ms] refreshing cache from master, [141ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [143ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@724461488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@549ed413 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2090739197 after retry RPC@1445878803) -> (continuation of Deferred@968441553 after retry RPC@1575500337) -> (continuation of Deferred@1497593398 after retry RPC@1172760221), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d3ef481 -> wakeup thread Time-limited test -> (continuation of Deferred@2090739197 after retry RPC@1445878803) -> (continuation of Deferred@968441553 after retry RPC@1575500337) -> (continuation of Deferred@1497593398 after retry RPC@1172760221))) after lookup
07:57:06.112 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:57:06.113 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.113 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.113 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.391 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:06.391 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)[L],137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:06.392 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=424), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [2ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [23ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [25ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [28ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [28ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [43ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [45ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [46ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [46ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [83ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [84ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [87ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [87ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [141ms] refreshing cache from master, [141ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [143ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [145ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [146ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [146ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [422ms] refreshing cache from master, [422ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [424ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@724461488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@549ed413 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2090739197 after retry RPC@1445878803) -> (continuation of Deferred@968441553 after retry RPC@1575500337) -> (continuation of Deferred@1497593398 after retry RPC@1172760221) -> (continuation of Deferred@1468360829 after retry RPC@1759308365), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d3ef481 -> wakeup thread Time-limited test -> (continuation of Deferred@2090739197 after retry RPC@1445878803) -> (continuation of Deferred@968441553 after retry RPC@1575500337) -> (continuation of Deferred@1497593398 after retry RPC@1172760221) -> (continuation of Deferred@1468360829 after retry RPC@1759308365))) after lookup
07:57:06.392 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] connecting to peer
07:57:06.395 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.395 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] cleaning up while in state CONNECTING due to: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:06.395 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783
07:57:07.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.224784  6864 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:57:07.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.224905  6864 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:07.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.225484  6864 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:07.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:07.225998  6737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111)
07:57:07.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:07.226505  6737 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783): Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111)
07:57:07.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.229085  6633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "137d6728589143c1a8ae1e4cc280df77" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "f772946b4e6940dda07baac926a093bf" is_pre_election: true
07:57:07.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.229357  6740 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 137d6728589143c1a8ae1e4cc280df77; no voters: 12bc83bf6ff640bc9820d875747f49cc, f772946b4e6940dda07baac926a093bf
07:57:07.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.229480  6864 raft_consensus.cc:2726] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
07:57:07.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.319993  6869 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 12bc83bf6ff640bc9820d875747f49cc)
07:57:07.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.320101  6869 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:07.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.320444  6869 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:57:07.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:07.320883  6565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111)
07:57:07.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:07.321411  6565 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783): Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111)
07:57:07.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.324530  6805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "f772946b4e6940dda07baac926a093bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "137d6728589143c1a8ae1e4cc280df77" is_pre_election: true
07:57:07.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.324695  6805 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f772946b4e6940dda07baac926a093bf in term 8.
07:57:07.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.324944  6568 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 137d6728589143c1a8ae1e4cc280df77, f772946b4e6940dda07baac926a093bf; no voters: 12bc83bf6ff640bc9820d875747f49cc
07:57:07.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.325076  6869 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 9 FOLLOWER]: Leader pre-election won for term 10
07:57:07.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.325124  6869 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 9 FOLLOWER]: Starting leader election (detected failure of leader 12bc83bf6ff640bc9820d875747f49cc)
07:57:07.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.325150  6869 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 9 FOLLOWER]: Advancing to term 10
07:57:07.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.325701  6869 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:07.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.325852  6869 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 10 election: Requested vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:57:07.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.325968  6805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "f772946b4e6940dda07baac926a093bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "137d6728589143c1a8ae1e4cc280df77"
07:57:07.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.326063  6805 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 8 FOLLOWER]: Advancing to term 10
07:57:07.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:07.326372  6565 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783): Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111)
07:57:07.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.327265  6805 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f772946b4e6940dda07baac926a093bf in term 10.
07:57:07.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.327422  6568 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 137d6728589143c1a8ae1e4cc280df77, f772946b4e6940dda07baac926a093bf; no voters: 12bc83bf6ff640bc9820d875747f49cc
07:57:07.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.327551  6869 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 10 FOLLOWER]: Leader election won for term 10
07:57:07.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.327720  6869 raft_consensus.cc:686] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 10 LEADER]: Becoming Leader. State: Replica: f772946b4e6940dda07baac926a093bf, State: Running, Role: LEADER
07:57:07.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.327879  6869 consensus_queue.cc:237] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:07.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.328629  4839 catalog_manager.cc:5495] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf reported cstate change: term changed from 9 to 10, leader changed from 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65) to f772946b4e6940dda07baac926a093bf (127.4.157.67). New cstate: current_term: 10 leader_uuid: "f772946b4e6940dda07baac926a093bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } health_report { overall_health: HEALTHY } } }
07:57:07.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.705509  6805 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 10 FOLLOWER]: Refusing update from remote peer f772946b4e6940dda07baac926a093bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
07:57:07.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.705946  6869 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
07:57:07.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:07.707387  6886 mvcc.cc:205] Tried to move back new op lower bound from 7305205456126656512 to 7305205451654250496. Current Snapshot: MvccSnapshot[applied={T|T < 7305205456126656512}]
07:57:07.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:07.709514  6565 consensus_peers.cc:487] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf -> Peer 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783): Couldn't send request to peer 12bc83bf6ff640bc9820d875747f49cc. Status: Network error: Client connection negotiation failed: client connection to 127.4.157.65:34783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:57:08.190 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:08.190 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:57:08.191 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=2223), Traces: [0ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [2ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [2ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [23ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [25ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [28ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [28ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [43ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [45ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [46ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [46ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [83ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [84ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [87ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [87ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [141ms] refreshing cache from master, [141ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [143ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [145ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [146ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [146ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [422ms] refreshing cache from master, [422ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [424ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [425ms] sending RPC to server 12bc83bf6ff640bc9820d875747f49cc, [428ms] received response from server 12bc83bf6ff640bc9820d875747f49cc: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [428ms] delaying RPC due to: Network error: Failed to connect to peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783): Connection refused: /127.4.157.65:34783, [2221ms] refreshing cache from master, [2221ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [2223ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@724461488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@549ed413 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2090739197 after retry RPC@1445878803) -> (continuation of Deferred@968441553 after retry RPC@1575500337) -> (continuation of Deferred@1497593398 after retry RPC@1172760221) -> (continuation of Deferred@1468360829 after retry RPC@1759308365) -> (continuation of Deferred@1620303359 after retry RPC@1863507234), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5d3ef481 -> wakeup thread Time-limited test -> (continuation of Deferred@2090739197 after retry RPC@1445878803) -> (continuation of Deferred@968441553 after retry RPC@1575500337) -> (continuation of Deferred@1497593398 after retry RPC@1172760221) -> (continuation of Deferred@1468360829 after retry RPC@1759308365) -> (continuation of Deferred@1620303359 after retry RPC@1863507234))) after lookup
07:57:08.192 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:08.193 [DEBUG - kudu-nio-0] (Connection.java:757) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] Successfully connected to peer
07:57:08.208 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xd7c45e19, L:/127.0.0.1:36666 - R:/127.4.157.67:39629] using TOKEN/null
07:57:08.219 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.157.65:34783
07:57:08.219 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.220176  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/logs
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.65:34783
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.65
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:57:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40963
07:57:08.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.65
07:57:08.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:57:08.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:57:08.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:57:08.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:57:08.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:08.305361  6901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:57:08.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:08.305600  6901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:57:08.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:08.305631  6901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:57:08.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:08.306993  6901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:57:08.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:08.307070  6901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.65
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.308447  6901 tablet_server_runner.cc:78] Tablet server non-default flags:
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.65:34783
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:57:08.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.65
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40963
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.6901
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.65
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/logs
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:57:08.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.308707  6901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:57:08.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.309087  6901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:57:08.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:08.315177  6907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:08.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:08.315138  6909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:08.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.315392  6901 server_base.cc:1034] running on GCE node
07:57:08.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:08.315526  6906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:08.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.315775  6901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:57:08.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.315982  6901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:57:08.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.317173  6901 hybrid_clock.cc:648] HybridClock initialized: now 1783497428317155 us; error 41 us; skew 500 ppm
07:57:08.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.318692  6901 webserver.cc:460] Webserver started at http://127.4.157.65:40963/ using document root <none> and password file <none>
07:57:08.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.318995  6901 fs_manager.cc:359] Metadata directory not provided
07:57:08.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.319080  6901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:57:08.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.320513  6901 fs_manager.cc:709] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
07:57:08.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.321336  6915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:57:08.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.321511  6901 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.321595  6901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "12bc83bf6ff640bc9820d875747f49cc"
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.322031  6901 fs_report.cc:383] FS layout report
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/wal
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/data
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:57:08.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:57:08.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.333518  6901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:57:08.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.333850  6901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:57:08.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.334007  6901 kserver.cc:163] Server-wide thread pool size limit: 3276
07:57:08.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.334299  6901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:57:08.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.334822  6922 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
07:57:08.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.336311  6901 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:57:08.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.336390  6901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s	user 0.000s	sys 0.000s
07:57:08.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.336426  6901 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
07:57:08.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.337211  6901 ts_tablet_manager.cc:610] Registered 1 tablets
07:57:08.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.337263  6901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s	user 0.001s	sys 0.000s
07:57:08.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.337330  6922 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Bootstrap starting.
07:57:08.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.344558  6901 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.65:34783
07:57:08.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.344666  7029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.65:34783 every 8 connection(s)
07:57:08.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.344993  6901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-0/data/info.pb
07:57:08.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.346465  6922 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Log is configured to *not* fsync() on all Append() calls
07:57:08.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.347915  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 6901
07:57:08.348 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:57:08.352 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:57:08.353 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.157.67:39629
07:57:08.353 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:57:08.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.353593  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 6550
07:57:08.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.353632  6922 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:57:08.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.353933  6922 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Bootstrap complete.
07:57:08.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.355406  6922 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.005s
07:57:08.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.355670  7030 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:57:08.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.355785  7030 heartbeater.cc:464] Registering TS with master...
07:57:08.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.355983  7030 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:57:08.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.356912  6922 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:08.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.357313  6922 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bc83bf6ff640bc9820d875747f49cc, State: Initialized, Role: FOLLOWER
07:57:08.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.357465  6922 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:08.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.357779  6922 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
07:57:08.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.358083  7032 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:57:08.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.358141  7032 heartbeater.cc:464] Registering TS with master...
07:57:08.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.358212  7032 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:57:08.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.358227  4911 ts_manager.cc:194] Re-registered known tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:57:08.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.358686  4839 ts_manager.cc:194] Re-registered known tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:57:08.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.359233  4839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.65:35589
07:57:08.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.359723  7031 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:57:08.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.359779  7031 heartbeater.cc:464] Registering TS with master...
07:57:08.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.360090  7031 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:57:08.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.360176  7032 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:57:08.360 [DEBUG - kudu-nio-0] (Connection.java:239) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] handling channelInactive
07:57:08.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:08.360538  4766 ts_manager.cc:194] Re-registered known tserver with Master: 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783)
07:57:08.360 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state READY due to: connection closed
07:57:08.361 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:57:08.362 [DEBUG - Time-limited test] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:08.363 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.363 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.363 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.371 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:08.371 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:57:08.371 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [1ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [1ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@614559679(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@4390af68 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c9ff794 -> wakeup thread Time-limited test)) after lookup
07:57:08.371 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:08.372 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.372 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.372 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.390 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:08.390 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:57:08.391 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [1ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [1ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [9ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [10ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [11ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@614559679(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@4390af68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@813880237 after retry RPC@1736743874), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c9ff794 -> wakeup thread Time-limited test -> (continuation of Deferred@813880237 after retry RPC@1736743874))) after lookup
07:57:08.391 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:08.393 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.393 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.393 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.431 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:08.431 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:57:08.432 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [1ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [1ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [9ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [10ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [11ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [29ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [31ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [31ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [69ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@614559679(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@4390af68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@813880237 after retry RPC@1736743874) -> (continuation of Deferred@1910148491 after retry RPC@851657465), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c9ff794 -> wakeup thread Time-limited test -> (continuation of Deferred@813880237 after retry RPC@1736743874) -> (continuation of Deferred@1910148491 after retry RPC@851657465))) after lookup
07:57:08.432 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:08.433 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.433 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.433 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.551 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:08.551 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:57:08.551 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=189), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [1ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [1ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [9ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [10ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [11ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [29ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [31ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [31ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [69ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [70ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [71ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [71ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [188ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@614559679(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@4390af68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@813880237 after retry RPC@1736743874) -> (continuation of Deferred@1910148491 after retry RPC@851657465) -> (continuation of Deferred@1989084136 after retry RPC@1322693962), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c9ff794 -> wakeup thread Time-limited test -> (continuation of Deferred@813880237 after retry RPC@1736743874) -> (continuation of Deferred@1910148491 after retry RPC@851657465) -> (continuation of Deferred@1989084136 after retry RPC@1322693962))) after lookup
07:57:08.552 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:08.554 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.554 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.554 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.630 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:08.631 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353),f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)[L]]}]
07:57:08.631 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=269), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [1ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [1ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [9ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [10ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [11ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [29ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [31ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [31ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [69ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [70ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [71ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [71ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [188ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [190ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [192ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [192ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [266ms] refreshing cache from master, [266ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [268ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@614559679(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@4390af68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@813880237 after retry RPC@1736743874) -> (continuation of Deferred@1910148491 after retry RPC@851657465) -> (continuation of Deferred@1989084136 after retry RPC@1322693962) -> (continuation of Deferred@2120293519 after retry RPC@98999020), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c9ff794 -> wakeup thread Time-limited test -> (continuation of Deferred@813880237 after retry RPC@1736743874) -> (continuation of Deferred@1910148491 after retry RPC@851657465) -> (continuation of Deferred@1989084136 after retry RPC@1322693962) -> (continuation of Deferred@2120293519 after retry RPC@98999020))) after lookup
07:57:08.632 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] connecting to peer
07:57:08.633 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.633 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] cleaning up while in state CONNECTING due to: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:08.633 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f772946b4e6940dda07baac926a093bf(127.4.157.67:39629) for tablet 85e494f8d75243559db1622f01fa36f7: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629
07:57:09.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.719635  7043 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f772946b4e6940dda07baac926a093bf)
07:57:09.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.719751  7043 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:09.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.719974  7043 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:09.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:09.720891  6740 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:57:09.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.721318  7048 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:57:09.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.721437  7048 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:09.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.721777  7048 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:09.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:09.722378  6919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:57:09.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:09.722968  6919 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:57:09.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.724673  6984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "137d6728589143c1a8ae1e4cc280df77" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "12bc83bf6ff640bc9820d875747f49cc" is_pre_election: true
07:57:09.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.724818  6984 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137d6728589143c1a8ae1e4cc280df77 in term 9.
07:57:09.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.725126  6737 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, 137d6728589143c1a8ae1e4cc280df77; no voters: f772946b4e6940dda07baac926a093bf
07:57:09.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.725245  7043 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 10 FOLLOWER]: Leader pre-election won for term 11
07:57:09.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.725323  7043 raft_consensus.cc:482] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f772946b4e6940dda07baac926a093bf)
07:57:09.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.725348  7043 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 10 FOLLOWER]: Advancing to term 11
07:57:09.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.726049  7043 raft_consensus.cc:504] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:09.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.726219  7043 leader_election.cc:288] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 11 election: Requested vote from peers 12bc83bf6ff640bc9820d875747f49cc (127.4.157.65:34783), f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:09.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.726352  6984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "137d6728589143c1a8ae1e4cc280df77" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "12bc83bf6ff640bc9820d875747f49cc"
07:57:09.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.726432  6984 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 9 FOLLOWER]: Advancing to term 11
07:57:09.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.726464  6805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e494f8d75243559db1622f01fa36f7" candidate_uuid: "12bc83bf6ff640bc9820d875747f49cc" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "137d6728589143c1a8ae1e4cc280df77" is_pre_election: true
07:57:09.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.726574  6805 raft_consensus.cc:2354] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 12bc83bf6ff640bc9820d875747f49cc for earlier term 10. Current term is 11.
07:57:09.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:09.726804  6740 leader_election.cc:334] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111)
07:57:09.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.726934  6919 leader_election.cc:398] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 10 pre-election: Vote denied by peer 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353) with higher term. Message: Invalid argument: T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 12bc83bf6ff640bc9820d875747f49cc for earlier term 10. Current term is 11.
07:57:09.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.726976  6919 leader_election.cc:401] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
07:57:09.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.727706  6984 raft_consensus.cc:2454] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137d6728589143c1a8ae1e4cc280df77 in term 11.
07:57:09.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.727840  7048 raft_consensus.cc:2726] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66:34353) with higher term. Message: Invalid argument: T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 12bc83bf6ff640bc9820d875747f49cc for earlier term 10. Current term is 11.
07:57:09.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.727926  6737 leader_election.cc:302] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bc83bf6ff640bc9820d875747f49cc, 137d6728589143c1a8ae1e4cc280df77; no voters: f772946b4e6940dda07baac926a093bf
07:57:09.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.728024  7043 raft_consensus.cc:2781] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 11 FOLLOWER]: Leader election won for term 11
07:57:09.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.728163  7043 raft_consensus.cc:686] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [term 11 LEADER]: Becoming Leader. State: Replica: 137d6728589143c1a8ae1e4cc280df77, State: Running, Role: LEADER
07:57:09.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.728243  7043 consensus_queue.cc:237] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:09.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:09.728967  4839 catalog_manager.cc:5495] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 reported cstate change: term changed from 10 to 11, leader changed from f772946b4e6940dda07baac926a093bf (127.4.157.67) to 137d6728589143c1a8ae1e4cc280df77 (127.4.157.66). New cstate: current_term: 11 leader_uuid: "137d6728589143c1a8ae1e4cc280df77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } health_report { overall_health: UNKNOWN } } }
07:57:10.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.246716  6984 raft_consensus.cc:1261] T 85e494f8d75243559db1622f01fa36f7 P 12bc83bf6ff640bc9820d875747f49cc [term 11 FOLLOWER]: Refusing update from remote peer 137d6728589143c1a8ae1e4cc280df77: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
07:57:10.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.247180  7066 consensus_queue.cc:1035] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
07:57:10.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.248657  7068 mvcc.cc:205] Tried to move back new op lower bound from 7305205465951842304 to 7305205462335504384. Current Snapshot: MvccSnapshot[applied={T|T < 7305205465951842304}]
07:57:10.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:10.250877  6740 consensus_peers.cc:487] T 85e494f8d75243559db1622f01fa36f7 P 137d6728589143c1a8ae1e4cc280df77 -> Peer f772946b4e6940dda07baac926a093bf (127.4.157.67:39629): Couldn't send request to peer f772946b4e6940dda07baac926a093bf. Status: Network error: Client connection negotiation failed: client connection to 127.4.157.67:39629: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:57:10.611 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 85e494f8d75243559db1622f01fa36f7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776' with partition [<start>, <end>)
07:57:10.612 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=85e494f8d75243559db1622f01fa36f7@[12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783),137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)[L],f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)]}]
07:57:10.612 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=85e494f8d75243559db1622f01fa36f7, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2250), Traces: [0ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [1ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [1ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [9ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [10ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [11ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [29ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [31ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [31ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [69ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [70ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [71ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [71ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [188ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [190ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [192ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [192ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [266ms] refreshing cache from master, [266ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [268ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, [269ms] sending RPC to server f772946b4e6940dda07baac926a093bf, [271ms] received response from server f772946b4e6940dda07baac926a093bf: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [271ms] delaying RPC due to: Network error: Failed to connect to peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629): Connection refused: /127.4.157.67:39629, [2247ms] refreshing cache from master, [2247ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.157.125:45997, [2249ms] Sub RPC GetTableLocations: received response from server master-127.4.157.125:45997: OK, deferred=Deferred@614559679(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/524368637@4390af68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@813880237 after retry RPC@1736743874) -> (continuation of Deferred@1910148491 after retry RPC@851657465) -> (continuation of Deferred@1989084136 after retry RPC@1322693962) -> (continuation of Deferred@2120293519 after retry RPC@98999020) -> (continuation of Deferred@2082165041 after retry RPC@2113489917), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4c9ff794 -> wakeup thread Time-limited test -> (continuation of Deferred@813880237 after retry RPC@1736743874) -> (continuation of Deferred@1910148491 after retry RPC@851657465) -> (continuation of Deferred@1989084136 after retry RPC@1322693962) -> (continuation of Deferred@2120293519 after retry RPC@98999020) -> (continuation of Deferred@2082165041 after retry RPC@2113489917))) after lookup
07:57:10.613 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] connecting to peer
07:57:10.615 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] Successfully connected to peer
07:57:10.633 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x9eaa0c35, L:/127.0.0.1:36178 - R:/127.4.157.66:34353] using TOKEN/null
07:57:10.642 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.157.67:39629
07:57:10.642 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:57:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.643105  4725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:57:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskJhaYq9/build/release/bin/kudu
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/logs
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.67:39629
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.67
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46573
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.67
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:57:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:57:10.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:57:10.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:10.726620  7079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
07:57:10.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:10.726907  7079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:57:10.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:10.726933  7079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
07:57:10.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:10.728091  7079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
07:57:10.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:10.728139  7079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.157.67
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.729480  7079 tablet_server_runner.cc:78] Tablet server non-default flags:
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.157.84:43769
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.157.67:39629
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.157.67
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46573
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.157.126:38135,127.4.157.125:45997,127.4.157.124:46883
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.7079
07:57:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.157.67
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/logs
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 06:42:21 UTC on ccycloud.quasar-lnvdon.root.comops.site
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492842
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.729679  7079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:57:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.729929  7079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
07:57:10.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:10.735077  7085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:10.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:10.735131  7084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:10.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.735347  7079 server_base.cc:1034] running on GCE node
07:57:10.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 07:57:10.735077  7087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:57:10.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.736025  7079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:57:10.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.736292  7079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:57:10.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.737428  7079 hybrid_clock.cc:648] HybridClock initialized: now 1783497430737414 us; error 28 us; skew 500 ppm
07:57:10.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.738658  7079 webserver.cc:460] Webserver started at http://127.4.157.67:46573/ using document root <none> and password file <none>
07:57:10.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.738904  7079 fs_manager.cc:359] Metadata directory not provided
07:57:10.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.738946  7079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:57:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.740232  7079 fs_manager.cc:709] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
07:57:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.740926  7093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:57:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.741190  7079 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
07:57:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.741289  7079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data,/tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:57:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f772946b4e6940dda07baac926a093bf"
07:57:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 07:56:44 on dist-test-slave-5j59"
07:57:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.741678  7079 fs_report.cc:383] FS layout report
07:57:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:57:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/wal
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/data
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:57:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:57:10.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.757598  7079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
07:57:10.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.757972  7079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:57:10.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.758198  7079 kserver.cc:163] Server-wide thread pool size limit: 3276
07:57:10.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.758486  7079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
07:57:10.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.758980  7100 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
07:57:10.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.760281  7079 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:57:10.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.760354  7079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
07:57:10.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.760393  7079 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
07:57:10.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.761113  7079 ts_tablet_manager.cc:610] Registered 1 tablets
07:57:10.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.761207  7079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s	user 0.001s	sys 0.000s
07:57:10.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.761253  7100 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap starting.
07:57:10.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.768190  7079 rpc_server.cc:307] RPC server started. Bound to: 127.4.157.67:39629
07:57:10.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.768306  7207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.157.67:39629 every 8 connection(s)
07:57:10.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.768616  7079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJhaYq9/test-tmp/mini-kudu-cluster7966722838338850321/ts-2/data/info.pb
07:57:10.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.770437  7100 log.cc:824] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Log is configured to *not* fsync() on all Append() calls
07:57:10.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.775106  7209 heartbeater.cc:347] Connected to a master server at 127.4.157.126:38135
07:57:10.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.775205  7209 heartbeater.cc:464] Registering TS with master...
07:57:10.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.775214  7210 heartbeater.cc:347] Connected to a master server at 127.4.157.125:45997
07:57:10.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.775275  7210 heartbeater.cc:464] Registering TS with master...
07:57:10.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.775403  7210 heartbeater.cc:511] Master 127.4.157.125:45997 requested a full tablet report, sending...
07:57:10.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.775403  7209 heartbeater.cc:511] Master 127.4.157.126:38135 requested a full tablet report, sending...
07:57:10.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.776046  4767 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:10.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.776036  4838 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:10.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.776494  4838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.157.67:44901
07:57:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.777072  7208 heartbeater.cc:347] Connected to a master server at 127.4.157.124:46883
07:57:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.777127  7208 heartbeater.cc:464] Registering TS with master...
07:57:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.777180  7208 heartbeater.cc:511] Master 127.4.157.124:46883 requested a full tablet report, sending...
07:57:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.777359  7100 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:57:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.777482  4910 ts_manager.cc:194] Re-registered known tserver with Master: f772946b4e6940dda07baac926a093bf (127.4.157.67:39629)
07:57:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.777674  7100 tablet_bootstrap.cc:492] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Bootstrap complete.
07:57:10.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.778069  4725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJhaYq9/build/release/bin/kudu as pid 7079
07:57:10.778 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:57:10.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.778656  7100 ts_tablet_manager.cc:1397] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.010s
07:57:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.779799  7100 raft_consensus.cc:348] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.780102  7100 raft_consensus.cc:729] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: f772946b4e6940dda07baac926a093bf, State: Initialized, Role: FOLLOWER
07:57:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.780230  7100 consensus_queue.cc:260] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137d6728589143c1a8ae1e4cc280df77" member_type: VOTER last_known_addr { host: "127.4.157.66" port: 34353 } } peers { permanent_uuid: "12bc83bf6ff640bc9820d875747f49cc" member_type: VOTER last_known_addr { host: "127.4.157.65" port: 34783 } } peers { permanent_uuid: "f772946b4e6940dda07baac926a093bf" member_type: VOTER last_known_addr { host: "127.4.157.67" port: 39629 } }
07:57:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.780457  7100 ts_tablet_manager.cc:1428] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
07:57:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.780575  7210 heartbeater.cc:503] Master 127.4.157.125:45997 was elected leader, sending a full tablet report...
07:57:10.782 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:57:10.785 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497404776, tablet=85e494f8d75243559db1622f01fa36f7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:57:10.790 [DEBUG - main] (Connection.java:575) [peer master-127.4.157.126:38135(127.4.157.126:38135)] disconnecting while in state READY
07:57:10.791 [DEBUG - main] (Connection.java:575) [peer 12bc83bf6ff640bc9820d875747f49cc(127.4.157.65:34783)] disconnecting while in state TERMINATED
07:57:10.791 [DEBUG - main] (Connection.java:575) [peer f772946b4e6940dda07baac926a093bf(127.4.157.67:39629)] disconnecting while in state TERMINATED
07:57:10.794 [DEBUG - main] (Connection.java:575) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] disconnecting while in state READY
07:57:10.794 [DEBUG - main] (Connection.java:575) [peer master-127.4.157.124:46883(127.4.157.124:46883)] disconnecting while in state READY
07:57:10.794 [DEBUG - main] (Connection.java:575) [peer master-127.4.157.125:45997(127.4.157.125:45997)] disconnecting while in state READY
07:57:10.796 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.4.157.126:38135(127.4.157.126:38135)] handling channelInactive
07:57:10.796 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2704) Releasing all remaining resources
07:57:10.796 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.4.157.126:38135(127.4.157.126:38135)] cleaning up while in state READY due to: connection closed
07:57:10.797 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] handling channelInactive
07:57:10.797 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 137d6728589143c1a8ae1e4cc280df77(127.4.157.66:34353)] cleaning up while in state READY due to: connection closed
07:57:10.800 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.4.157.124:46883(127.4.157.124:46883)] handling channelInactive
07:57:10.800 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.4.157.124:46883(127.4.157.124:46883)] cleaning up while in state READY due to: connection closed
07:57:10.801 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.4.157.125:45997(127.4.157.125:45997)] handling channelInactive
07:57:10.801 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.4.157.125:45997(127.4.157.125:45997)] cleaning up while in state READY due to: connection closed
07:57:10.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.802249  7162 raft_consensus.cc:3037] T 85e494f8d75243559db1622f01fa36f7 P f772946b4e6940dda07baac926a093bf [term 10 FOLLOWER]: Advancing to term 11
07:57:10.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.802635  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 6901
07:57:10.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.808985  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 6722
07:57:10.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.809897  7223 mvcc.cc:205] Tried to move back new op lower bound from 7305205475900235776 to 7305205472167452672. Current Snapshot: MvccSnapshot[applied={T|T < 7305205475876941824}]
07:57:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.815653  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 7079
07:57:10.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.820988  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 4736
07:57:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.826140  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 4808
07:57:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 07:57:10.831648  4725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJhaYq9/build/release/bin/kudu with pid 4879
07:57:10.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T07:57:10Z chronyd exiting

Time: 28.714

OK (1 test)

