JUnit version 4.13.2
.05:22:33.212 [INFO - main] (RandomUtils.java:49) Using random seed: 1783488153207
05:22:33.255 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
05:22:33.307 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
05:22:33.323 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskarRtQd/test-tmp
05:22:33.326 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
05:22:33.330 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskarRtQd/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
05:22:33.615 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

05:22:33.770 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
05:22:33.771 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

05:22:33.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:33.775242 14696 mini_oidc.cc:152] Starting JWKS server
05:22:33.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:33.778976 14696 webserver.cc:460] Webserver started at http://127.0.0.1:46291/ using document root <none> and password file <none>
05:22:33.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:33.779439 14696 mini_oidc.cc:189] Starting OIDC Discovery server
05:22:33.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:33.781543 14696 webserver.cc:460] Webserver started at http://127.0.0.1:40277/ using document root <none> and password file <none>
05:22:33.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T05:22:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
05:22:33.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T05:22:33Z Disabled control of system clock
05:22:33.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:33.834762 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:33.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:33.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/wal
05:22:33.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/data
05:22:33.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:22:33.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:22:33.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:22:33.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:33.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:33.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:33.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/logs
05:22:33.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/data/info.pb
05:22:33.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:33.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:22:33.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:22:33.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:22:33.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:22:33.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:33.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:33.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:33.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
05:22:33.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:22:33.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:22:33.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:22:33.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.62:44495
05:22:33.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.62
05:22:33.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:33.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:33.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:33.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:33.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:22:33.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:22:33.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:33.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:22:33.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:46291/jwks.json with env {}
05:22:34.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.147166 14708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:22:34.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.147889 14708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:22:34.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.148523 14708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:22:34.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.181291 14708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
05:22:34.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.181617 14708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:22:34.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.181810 14708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
05:22:34.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.182089 14708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
05:22:34.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.182291 14708 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
05:22:34.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.182423 14708 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:46291/jwks.json
05:22:34.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:34.220734 14708 master_runner.cc:386] Master server non-default flags:
05:22:34.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:34.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:34.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:34.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:22:34.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/data
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/wal
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.62:44495
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:46291/jwks.json
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:34.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/data/info.pb
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.62
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/logs
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:22:34.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:22:34.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:22:34.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:22:34.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:22:34.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:22:34.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:34.222043 14708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:22:34.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:34.224244 14708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:22:34.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.243258 14714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:35.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.243340 14715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:35.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:34.243346 14717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:35.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:34.245110 14708 server_base.cc:1034] running on GCE node
05:22:35.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.437147 14708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:22:35.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.439862 14708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:22:35.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.441301 14708 hybrid_clock.cc:648] HybridClock initialized: now 1783488155441237 us; error 67 us; skew 500 ppm
05:22:35.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.442196 14708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:22:35.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.453830 14708 webserver.cc:460] Webserver started at http://127.14.90.62:46037/ using document root <none> and password file <none>
05:22:35.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.454879 14708 fs_manager.cc:359] Metadata directory not provided
05:22:35.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.455081 14708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:22:35.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.455498 14708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:22:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.460513 14708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/data/instance:
05:22:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ee4c72df004c47f3b7df59ebbe16d0d0"
05:22:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:35 on dist-test-slave-zq0g"
05:22:35.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.461681 14708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/wal/instance:
05:22:35.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ee4c72df004c47f3b7df59ebbe16d0d0"
05:22:35.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:35 on dist-test-slave-zq0g"
05:22:35.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.469061 14708 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.005s	sys 0.004s
05:22:35.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.475059 14724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:22:35.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.476253 14708 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
05:22:35.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.476568 14708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/wal
05:22:35.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ee4c72df004c47f3b7df59ebbe16d0d0"
05:22:35.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:35 on dist-test-slave-zq0g"
05:22:35.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.476967 14708 fs_report.cc:383] FS layout report
05:22:35.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:22:35.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/wal
05:22:35.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/wal
05:22:35.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/data/data
05:22:35.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:22:35.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:22:35.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:22:35.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:22:35.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:22:35.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:22:35.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:22:35.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:22:35.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:22:35.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:22:35.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:22:35.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.518095 14708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:22:35.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.519668 14708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:22:35.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.520102 14708 kserver.cc:163] Server-wide thread pool size limit: 3276
05:22:35.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.593539 14708 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.62:44495
05:22:35.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.593677 14775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.62:44495 every 8 connection(s)
05:22:35.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.596652 14708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/data/info.pb
05:22:35.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.601907 14776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
05:22:35.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.606360 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 14708
05:22:35.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.606843 14696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-0/wal/instance
05:22:35.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.610311 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/wal
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/data
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/logs
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/data/info.pb
05:22:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:22:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:22:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:22:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:22:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
05:22:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:22:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:22:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:22:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.61:34695
05:22:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.61
05:22:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:22:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:22:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:22:35.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:46291/jwks.json with env {}
05:22:35.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.615348 14776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } has no permanent_uuid. Determining permanent_uuid...
05:22:35.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.651134 14776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } has no permanent_uuid. Determining permanent_uuid...
05:22:35.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.653566 14725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.61:34695: connect: Connection refused (error 111)
05:22:35.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.657953 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.61:34695: Network error: Client connection negotiation failed: client connection to 127.14.90.61:34695: connect: Connection refused (error 111)
05:22:35.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.707563 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } attempt: 1
05:22:35.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.712029 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.61:34695: Network error: Client connection negotiation failed: client connection to 127.14.90.61:34695: connect: Connection refused (error 111)
05:22:35.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.780480 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } attempt: 2
05:22:35.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.784386 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.61:34695: Network error: Client connection negotiation failed: client connection to 127.14.90.61:34695: connect: Connection refused (error 111)
05:22:35.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.875838 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } attempt: 3
05:22:35.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.880300 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.61:34695: Network error: Client connection negotiation failed: client connection to 127.14.90.61:34695: connect: Connection refused (error 111)
05:22:35.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.929492 14778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:22:35.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.930039 14778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:22:35.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.930670 14778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:22:35.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.962461 14778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
05:22:35.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.962772 14778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:22:35.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.962960 14778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
05:22:35.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.963150 14778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
05:22:35.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.963326 14778 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
05:22:35.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:35.963493 14778 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:46291/jwks.json
05:22:35.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.998356 14778 master_runner.cc:386] Master server non-default flags:
05:22:35.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:35.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:35.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:35.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:22:35.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/data
05:22:35.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/wal
05:22:35.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:22:35.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:35.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:35.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:35.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.61:34695
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:46291/jwks.json
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/data/info.pb
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.61
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:22:36.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:22:36.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:36.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/logs
05:22:36.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:36.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:22:36.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:22:36.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
05:22:36.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:22:36.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:22:36.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:22:36.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:22:36.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:22:36.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:22:36.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:35.999738 14778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:22:36.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:36.001266 14778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:22:36.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:36.020851 14788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:36.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:36.023756 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } attempt: 4
05:22:36.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:36.027383 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.61:34695: Network error: Client connection negotiation failed: client connection to 127.14.90.61:34695: connect: Connection refused (error 111)
05:22:36.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:36.326901 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } attempt: 5
05:22:36.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:36.331941 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.61:34695: Network error: Client connection negotiation failed: client connection to 127.14.90.61:34695: connect: Connection refused (error 111)
05:22:36.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:36.879419 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } attempt: 6
05:22:36.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:36.887408 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.61:34695: Network error: Client connection negotiation failed: client connection to 127.14.90.61:34695: connect: Connection refused (error 111)
05:22:37.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:36.020859 14790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:37.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:36.021745 14787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:37.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:36.022593 14778 server_base.cc:1034] running on GCE node
05:22:37.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.241257 14778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:22:37.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.244882 14778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:22:37.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.246385 14778 hybrid_clock.cc:648] HybridClock initialized: now 1783488157246298 us; error 85 us; skew 500 ppm
05:22:37.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.247439 14778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:22:37.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.254544 14778 webserver.cc:460] Webserver started at http://127.14.90.61:40771/ using document root <none> and password file <none>
05:22:37.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.255748 14778 fs_manager.cc:359] Metadata directory not provided
05:22:37.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.255985 14778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:22:37.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.256390 14778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:22:37.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.261466 14778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/data/instance:
05:22:37.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4172bc0e0659485a9670519b42b09eb5"
05:22:37.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:37 on dist-test-slave-zq0g"
05:22:37.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.262862 14778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/wal/instance:
05:22:37.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4172bc0e0659485a9670519b42b09eb5"
05:22:37.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:37 on dist-test-slave-zq0g"
05:22:37.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.270570 14778 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
05:22:37.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.276716 14799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:22:37.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.278074 14778 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.003s
05:22:37.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.278501 14778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/wal
05:22:37.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4172bc0e0659485a9670519b42b09eb5"
05:22:37.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:37 on dist-test-slave-zq0g"
05:22:37.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.278959 14778 fs_report.cc:383] FS layout report
05:22:37.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:22:37.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/wal
05:22:37.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/wal
05:22:37.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/data/data
05:22:37.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:22:37.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:22:37.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:22:37.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:22:37.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:22:37.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:22:37.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:22:37.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:22:37.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:22:37.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:22:37.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:22:37.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.319633 14778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:22:37.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.321319 14778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:22:37.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.321884 14778 kserver.cc:163] Server-wide thread pool size limit: 3276
05:22:37.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.394747 14778 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.61:34695
05:22:37.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.394840 14850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.61:34695 every 8 connection(s)
05:22:37.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.397800 14778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/data/info.pb
05:22:37.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.400362 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 14778
05:22:37.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.400884 14696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-1/wal/instance
05:22:37.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.403214 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:37.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:37.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/wal
05:22:37.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/data
05:22:37.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:22:37.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:22:37.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/logs
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/data/info.pb
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:22:37.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.60:44831
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.60
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:46291/jwks.json with env {}
05:22:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.403960 14851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
05:22:37.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.420439 14851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } has no permanent_uuid. Determining permanent_uuid...
05:22:37.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.445307 14851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } has no permanent_uuid. Determining permanent_uuid...
05:22:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.463210 14851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } has no permanent_uuid. Determining permanent_uuid...
05:22:37.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.465848 14803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:37.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.470410 14851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:37.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.520010 14851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 1
05:22:37.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.524366 14851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:37.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.592840 14851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 2
05:22:37.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.597741 14851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:37.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.689262 14851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 3
05:22:37.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.693432 14851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:37.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.730357 14853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:22:37.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.730939 14853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:22:37.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.731514 14853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:22:37.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.762853 14853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
05:22:37.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.763154 14853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:22:37.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.763345 14853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
05:22:37.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.763527 14853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
05:22:37.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.763701 14853 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
05:22:37.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.763844 14853 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:46291/jwks.json
05:22:37.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.799827 14853 master_runner.cc:386] Master server non-default flags:
05:22:37.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:37.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:37.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:37.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:22:37.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/data
05:22:37.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/wal
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.60:44831
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:46291/jwks.json
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/data/info.pb
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.60
05:22:37.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:22:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:22:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:22:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/logs
05:22:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:22:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:22:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
05:22:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:22:37.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:22:37.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:22:37.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:22:37.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:22:37.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:22:37.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.801187 14853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:22:37.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.803149 14853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:22:37.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.824646 14863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:37.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.836910 14851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 4
05:22:37.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.841148 14851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:37.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.947960 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } attempt: 7
05:22:37.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.964781 14776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } has no permanent_uuid. Determining permanent_uuid...
05:22:37.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.970579 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:38.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:38.029258 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 1
05:22:38.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:38.033763 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:38.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:38.115447 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 2
05:22:38.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:38.120018 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:38.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:38.140590 14851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 5
05:22:38.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:38.145080 14851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:38.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:38.205667 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 3
05:22:38.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:38.211297 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:38.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:38.351775 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 4
05:22:38.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:38.356627 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:38.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:38.640105 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 5
05:22:38.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:38.644932 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:38.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:38.692516 14851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 6
05:22:38.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:38.699692 14851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:39.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.824664 14866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:39.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:37.824827 14864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:39.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:37.825858 14853 server_base.cc:1034] running on GCE node
05:22:39.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.058815 14853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:22:39.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.061668 14853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:22:39.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.063061 14853 hybrid_clock.cc:648] HybridClock initialized: now 1783488159063046 us; error 59 us; skew 500 ppm
05:22:39.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.063885 14853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:22:39.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.070369 14853 webserver.cc:460] Webserver started at http://127.14.90.60:36955/ using document root <none> and password file <none>
05:22:39.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.071386 14853 fs_manager.cc:359] Metadata directory not provided
05:22:39.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.071578 14853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:22:39.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.071962 14853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:22:39.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.077728 14853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/data/instance:
05:22:39.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "59011bf584aa4c849b024a6d46aa5835"
05:22:39.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:39 on dist-test-slave-zq0g"
05:22:39.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.079137 14853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/wal/instance:
05:22:39.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "59011bf584aa4c849b024a6d46aa5835"
05:22:39.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:39 on dist-test-slave-zq0g"
05:22:39.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.088272 14853 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.000s	sys 0.008s
05:22:39.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.094254 14877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:22:39.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.095562 14853 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
05:22:39.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.096004 14853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/wal
05:22:39.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "59011bf584aa4c849b024a6d46aa5835"
05:22:39.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:39 on dist-test-slave-zq0g"
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.096443 14853 fs_report.cc:383] FS layout report
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/wal
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/wal
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/data/data
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:22:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:22:39.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.137169 14853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:22:39.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.138836 14853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:22:39.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.139395 14853 kserver.cc:163] Server-wide thread pool size limit: 3276
05:22:39.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.197563 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 6
05:22:39.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:39.206001 14776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.90.60:44831: Network error: Client connection negotiation failed: client connection to 127.14.90.60:44831: connect: Connection refused (error 111)
05:22:39.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.213668 14853 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.60:44831
05:22:39.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.213779 14930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.60:44831 every 8 connection(s)
05:22:39.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.216572 14853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/data/info.pb
05:22:39.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.222121 14931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
05:22:39.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.223044 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 14853
05:22:39.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.223474 14696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/master-2/wal/instance
05:22:39.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.236896 14931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } has no permanent_uuid. Determining permanent_uuid...
05:22:39.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.269449 14931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } has no permanent_uuid. Determining permanent_uuid...
05:22:39.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.281844 14931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } has no permanent_uuid. Determining permanent_uuid...
05:22:39.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.299916 14931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835: Bootstrap starting.
05:22:39.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.306031 14931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835: Neither blocks nor log segments found. Creating new log.
05:22:39.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.307878 14931 log.cc:824] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835: Log is configured to *not* fsync() on all Append() calls
05:22:39.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.314152 14931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835: No bootstrap required, opened a new log
05:22:39.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.335229 14931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } }
05:22:39.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.335922 14931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:22:39.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.336186 14931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59011bf584aa4c849b024a6d46aa5835, State: Initialized, Role: FOLLOWER
05:22:39.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.337090 14931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } }
05:22:39.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.339888 14940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } } }
05:22:39.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.340813 14940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [sys.catalog]: This master's current role is: FOLLOWER
05:22:39.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.341257 14931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [sys.catalog]: configured and running, proceeding with master startup.
05:22:39.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:39.368191 14951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
05:22:39.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:39.368552 14951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
05:22:39.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.364573 14853 master_runner.cc:186] Error getting master registration for 127.14.90.62:44495: OK, instance_id { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" instance_seqno: 1783488155572336 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
05:22:39.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.392432 14853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
05:22:39.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:39.395546 14883 tablet.cc:2369] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835: Can't schedule compaction. Clean time has not been advanced past its initial value.
05:22:39.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.709458 14940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:22:39.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.710049 14940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } }
05:22:39.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.712720 14940 leader_election.cc:288] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee4c72df004c47f3b7df59ebbe16d0d0 (127.14.90.62:44495), 4172bc0e0659485a9670519b42b09eb5 (127.14.90.61:34695)
05:22:39.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.760090 14851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 7
05:22:39.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.788177 14851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5: Bootstrap starting.
05:22:39.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.794692 14851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5: Neither blocks nor log segments found. Creating new log.
05:22:39.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.797493 14851 log.cc:824] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5: Log is configured to *not* fsync() on all Append() calls
05:22:39.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.807196 14851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5: No bootstrap required, opened a new log
05:22:39.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.829857 14851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } }
05:22:39.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.831110 14851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:22:39.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.831434 14851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4172bc0e0659485a9670519b42b09eb5, State: Initialized, Role: FOLLOWER
05:22:39.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.832518 14851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } }
05:22:39.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.836292 14955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } } }
05:22:39.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.837617 14955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [sys.catalog]: This master's current role is: FOLLOWER
05:22:39.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.839787 14851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [sys.catalog]: configured and running, proceeding with master startup.
05:22:39.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.841126 14826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59011bf584aa4c849b024a6d46aa5835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4172bc0e0659485a9670519b42b09eb5" is_pre_election: true
05:22:39.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.842324 14826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59011bf584aa4c849b024a6d46aa5835 in term 0.
05:22:39.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.844121 14878 leader_election.cc:302] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4172bc0e0659485a9670519b42b09eb5, 59011bf584aa4c849b024a6d46aa5835; no voters: 
05:22:39.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.845254 14940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [term 0 FOLLOWER]: Leader pre-election won for term 1
05:22:39.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.845626 14940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
05:22:39.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.845964 14940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [term 0 FOLLOWER]: Advancing to term 1
05:22:39.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.853351 14940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } }
05:22:39.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.857610 14940 leader_election.cc:288] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [CANDIDATE]: Term 1 election: Requested vote from peers ee4c72df004c47f3b7df59ebbe16d0d0 (127.14.90.62:44495), 4172bc0e0659485a9670519b42b09eb5 (127.14.90.61:34695)
05:22:39.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.857702 14826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59011bf584aa4c849b024a6d46aa5835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4172bc0e0659485a9670519b42b09eb5"
05:22:39.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.858657 14826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [term 0 FOLLOWER]: Advancing to term 1
05:22:39.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.865669 14826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59011bf584aa4c849b024a6d46aa5835 in term 1.
05:22:39.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.867895 14878 leader_election.cc:302] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4172bc0e0659485a9670519b42b09eb5, 59011bf584aa4c849b024a6d46aa5835; no voters: 
05:22:39.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.869135 14940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [term 1 FOLLOWER]: Leader election won for term 1
05:22:39.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.877185 14940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [term 1 LEADER]: Becoming Leader. State: Replica: 59011bf584aa4c849b024a6d46aa5835, State: Running, Role: LEADER
05:22:39.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.879285 14940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } }
05:22:39.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.916244 14962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59011bf584aa4c849b024a6d46aa5835. Latest consensus state: current_term: 1 leader_uuid: "59011bf584aa4c849b024a6d46aa5835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } } }
05:22:39.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.923238 14962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [sys.catalog]: This master's current role is: LEADER
05:22:39.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.935669 14970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
05:22:39.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:39.936691 14968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
05:22:39.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:39.937332 14968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
05:22:39.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.953714 14970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
05:22:39.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.915807 14778 master_runner.cc:186] Error getting master registration for 127.14.90.62:44495: OK, instance_id { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" instance_seqno: 1783488155572336 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
05:22:39.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.966708 14778 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
05:22:39.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.976363 14826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [term 1 FOLLOWER]: Refusing update from remote peer 59011bf584aa4c849b024a6d46aa5835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
05:22:39.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:39.978857 14940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
05:22:40.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.021920 14955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59011bf584aa4c849b024a6d46aa5835. Latest consensus state: current_term: 1 leader_uuid: "59011bf584aa4c849b024a6d46aa5835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } } }
05:22:40.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.027657 14955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [sys.catalog]: This master's current role is: FOLLOWER
05:22:40.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.029335 14971 mvcc.cc:205] Tried to move back new op lower bound from 7305167503242440704 to 7305167502896881664. Current Snapshot: MvccSnapshot[applied={T|T < 7305167503242440704}]
05:22:40.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.047365 14962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59011bf584aa4c849b024a6d46aa5835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } } }
05:22:40.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.048122 14962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [sys.catalog]: This master's current role is: LEADER
05:22:40.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.052145 14970 catalog_manager.cc:1347] Generated new cluster ID: 1cca28c6956c45d79f52cfa04d95dd62
05:22:40.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.052524 14970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
05:22:40.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.059983 14955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59011bf584aa4c849b024a6d46aa5835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } } }
05:22:40.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.060866 14955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5 [sys.catalog]: This master's current role is: FOLLOWER
05:22:40.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.091182 14970 catalog_manager.cc:1370] Generated new certificate authority record
05:22:40.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.096230 14970 catalog_manager.cc:1504] Loading token signing keys...
05:22:40.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.135197 14970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835: Generated new TSK 0
05:22:40.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.137032 14970 catalog_manager.cc:1514] Initializing in-progress tserver states...
05:22:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.160928 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:22:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data
05:22:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:22:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:22:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:22:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:40.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:40.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/logs
05:22:40.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:22:40.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:40.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:22:40.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:22:40.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:22:40.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.1:0
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.1
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.1
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:22:40.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.239492 14776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } attempt: 7
05:22:40.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.336769 14776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0: Bootstrap starting.
05:22:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.356541 14776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0: Neither blocks nor log segments found. Creating new log.
05:22:40.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.369153 14776 log.cc:824] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0: Log is configured to *not* fsync() on all Append() calls
05:22:40.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.384155 14776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0: No bootstrap required, opened a new log
05:22:40.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.511675 14776 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } }
05:22:40.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.516685 14776 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:22:40.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.517412 14776 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4c72df004c47f3b7df59ebbe16d0d0, State: Initialized, Role: FOLLOWER
05:22:40.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.518888 14776 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } }
05:22:40.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.527525 14979 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } } }
05:22:40.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.533562 14776 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [sys.catalog]: configured and running, proceeding with master startup.
05:22:40.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.550688 14979 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [sys.catalog]: This master's current role is: FOLLOWER
05:22:40.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.533737 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59011bf584aa4c849b024a6d46aa5835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" is_pre_election: true
05:22:40.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:40.534863 14730 tablet.cc:2369] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
05:22:40.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.534611 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59011bf584aa4c849b024a6d46aa5835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0"
05:22:40.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.535897 14749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [term 0 FOLLOWER]: Advancing to term 1
05:22:40.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.570719 14749 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [term 1 FOLLOWER]: Refusing update from remote peer 59011bf584aa4c849b024a6d46aa5835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
05:22:40.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.579635 14979 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:22:40.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.580178 14979 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } }
05:22:40.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.581440 14962 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
05:22:40.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.589349 14826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4172bc0e0659485a9670519b42b09eb5" is_pre_election: true
05:22:40.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.590677 14904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59011bf584aa4c849b024a6d46aa5835" is_pre_election: true
05:22:40.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.593806 14728 leader_election.cc:302] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee4c72df004c47f3b7df59ebbe16d0d0; no voters: 4172bc0e0659485a9670519b42b09eb5, 59011bf584aa4c849b024a6d46aa5835
05:22:40.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.596499 14979 leader_election.cc:288] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4172bc0e0659485a9670519b42b09eb5 (127.14.90.61:34695), 59011bf584aa4c849b024a6d46aa5835 (127.14.90.60:44831)
05:22:40.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.653388 14984 mvcc.cc:205] Tried to move back new op lower bound from 7305167503242440704 to 7305167502896881664. Current Snapshot: MvccSnapshot[applied={T|T < 7305167503242440704}]
05:22:40.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.690876 14979 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
05:22:40.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.695950 14982 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59011bf584aa4c849b024a6d46aa5835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } } }
05:22:40.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.699501 14982 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [sys.catalog]: This master's current role is: FOLLOWER
05:22:40.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.698320 14981 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59011bf584aa4c849b024a6d46aa5835. Latest consensus state: current_term: 1 leader_uuid: "59011bf584aa4c849b024a6d46aa5835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } } }
05:22:40.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.700453 14981 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0 [sys.catalog]: This master's current role is: FOLLOWER
05:22:40.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.727546 14997 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
05:22:40.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:40.853129 14976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:22:40.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:40.853708 14976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:22:40.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:40.854313 14976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:22:40.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:40.887914 14976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:22:40.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:40.889032 14976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.1
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.925813 14976 tablet_server_runner.cc:78] Tablet server non-default flags:
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.1:0
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:22:40.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.1
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.1
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/logs
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:22:40.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:22:40.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:22:40.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.927399 14976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:22:40.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.929291 14976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:22:40.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.943213 14968 catalog_manager.cc:1259] Loaded cluster ID: 1cca28c6956c45d79f52cfa04d95dd62
05:22:40.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.943868 14968 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5: loading cluster ID for follower catalog manager: success
05:22:40.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.952116 14968 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5: acquiring CA information for follower catalog manager: success
05:22:40.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:40.958714 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:40.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:40.976084 14968 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4172bc0e0659485a9670519b42b09eb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
05:22:41.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:41.043658 15009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59011bf584aa4c849b024a6d46aa5835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4c72df004c47f3b7df59ebbe16d0d0" member_type: VOTER last_known_addr { host: "127.14.90.62" port: 44495 } } peers { permanent_uuid: "4172bc0e0659485a9670519b42b09eb5" member_type: VOTER last_known_addr { host: "127.14.90.61" port: 34695 } } peers { permanent_uuid: "59011bf584aa4c849b024a6d46aa5835" member_type: VOTER last_known_addr { host: "127.14.90.60" port: 44831 } } }
05:22:41.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:41.058713 15009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59011bf584aa4c849b024a6d46aa5835 [sys.catalog]: This master's current role is: LEADER
05:22:41.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:41.759096 14997 catalog_manager.cc:1259] Loaded cluster ID: 1cca28c6956c45d79f52cfa04d95dd62
05:22:41.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:41.759750 14997 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0: loading cluster ID for follower catalog manager: success
05:22:41.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:41.794888 14997 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0: acquiring CA information for follower catalog manager: success
05:22:41.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:41.801546 14997 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee4c72df004c47f3b7df59ebbe16d0d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
05:22:42.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:40.960101 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:42.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:40.967304 15007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:42.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.885080 14976 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
05:22:42.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:42.885568 15005 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1929 milliseconds
05:22:42.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.889485 14976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:22:42.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.892289 14976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:22:42.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.893738 14976 hybrid_clock.cc:648] HybridClock initialized: now 1783488162893675 us; error 78 us; skew 500 ppm
05:22:42.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.894730 14976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:22:42.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.908185 14976 webserver.cc:460] Webserver started at http://127.14.90.1:34663/ using document root <none> and password file <none>
05:22:42.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.909327 14976 fs_manager.cc:359] Metadata directory not provided
05:22:42.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.909775 14976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:22:42.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.910458 14976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:22:42.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.915989 14976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/instance:
05:22:42.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "969c70d1882541a4b320cca5f9979912"
05:22:42.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:42 on dist-test-slave-zq0g"
05:22:42.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.917824 14976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal/instance:
05:22:42.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "969c70d1882541a4b320cca5f9979912"
05:22:42.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:42 on dist-test-slave-zq0g"
05:22:42.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.927242 14976 fs_manager.cc:691] Time spent creating directory manager: real 0.009s	user 0.004s	sys 0.003s
05:22:42.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.935010 15017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:22:42.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.936410 14976 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.003s
05:22:42.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.936823 14976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:22:42.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "969c70d1882541a4b320cca5f9979912"
05:22:42.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:42 on dist-test-slave-zq0g"
05:22:42.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.937282 14976 fs_report.cc:383] FS layout report
05:22:42.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/data
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:22:42.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:22:42.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.984532 14976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:22:42.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.986183 14976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:22:42.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.986789 14976 kserver.cc:163] Server-wide thread pool size limit: 3276
05:22:42.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.990101 14976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:22:42.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.995349 14976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
05:22:42.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.995636 14976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
05:22:42.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.995995 14976 ts_tablet_manager.cc:610] Registered 0 tablets
05:22:42.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:42.996227 14976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
05:22:43.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.202672 14976 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.1:46843
05:22:43.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.202812 15129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.1:46843 every 8 connection(s)
05:22:43.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.205451 14976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:22:43.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.214152 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 14976
05:22:43.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.215001 14696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal/instance
05:22:43.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.230000 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:43.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:43.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:22:43.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data
05:22:43.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/logs
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.2:0
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.2
05:22:43.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.2
05:22:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:43.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:22:43.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.286739 15130 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:22:43.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.287285 15130 heartbeater.cc:464] Registering TS with master...
05:22:43.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.288558 15130 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:22:43.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.289498 15132 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:22:43.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.289855 15132 heartbeater.cc:464] Registering TS with master...
05:22:43.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.290699 15132 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:22:43.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.293111 14894 ts_manager.cc:194] Registered new tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:22:43.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.294602 14741 ts_manager.cc:194] Registered new tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:22:43.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.296200 14894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.1:41801
05:22:43.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.304484 15131 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:22:43.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.304889 15131 heartbeater.cc:464] Registering TS with master...
05:22:43.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.305680 15131 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:22:43.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.309024 14816 ts_manager.cc:194] Registered new tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:22:43.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:43.633140 15140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:22:43.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:43.633774 15140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:22:43.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:43.634881 15140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:22:43.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:43.690361 15140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:22:43.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:43.691757 15140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.2
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.739794 15140 tablet_server_runner.cc:78] Tablet server non-default flags:
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.2:0
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:22:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.2
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.2
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/logs
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:22:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.741386 15140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:22:43.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.743666 15140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:22:43.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:43.769825 15150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:44.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:44.302429 15130 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:22:44.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:43.769937 15147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:44.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:43.769869 15148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:43.771091 15140 server_base.cc:1034] running on GCE node
05:22:44.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:44.996222 15140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:22:44.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:44.998778 15140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:22:45.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.000281 15140 hybrid_clock.cc:648] HybridClock initialized: now 1783488165000225 us; error 93 us; skew 500 ppm
05:22:45.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.001178 15140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:22:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.007699 15140 webserver.cc:460] Webserver started at http://127.14.90.2:34839/ using document root <none> and password file <none>
05:22:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.008728 15140 fs_manager.cc:359] Metadata directory not provided
05:22:45.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.008988 15140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:22:45.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.009554 15140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:22:45.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.014452 15140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/instance:
05:22:45.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a2f24784906a4e2ebb7a4d8073e17074"
05:22:45.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:45 on dist-test-slave-zq0g"
05:22:45.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.015703 15140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal/instance:
05:22:45.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a2f24784906a4e2ebb7a4d8073e17074"
05:22:45.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:45 on dist-test-slave-zq0g"
05:22:45.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.023221 15140 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
05:22:45.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.028959 15159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:22:45.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.030014 15140 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
05:22:45.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.030372 15140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:22:45.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a2f24784906a4e2ebb7a4d8073e17074"
05:22:45.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:45 on dist-test-slave-zq0g"
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.030835 15140 fs_report.cc:383] FS layout report
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/data
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:22:45.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:22:45.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.066147 15140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:22:45.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.067970 15140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:22:45.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.068565 15140 kserver.cc:163] Server-wide thread pool size limit: 3276
05:22:45.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.071496 15140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:22:45.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.077811 15140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
05:22:45.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.078104 15140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
05:22:45.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.078411 15140 ts_tablet_manager.cc:610] Registered 0 tablets
05:22:45.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.078699 15140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.002s	sys 0.000s
05:22:45.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.249863 15140 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.2:45685
05:22:45.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.250013 15271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.2:45685 every 8 connection(s)
05:22:45.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.252831 15140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:22:45.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.258443 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 15140
05:22:45.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.259302 14696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal/instance
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.296859 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/logs
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:45.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.3:0
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.3
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.3
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:22:45.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.314924 15274 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:22:45.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.315433 15274 heartbeater.cc:464] Registering TS with master...
05:22:45.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.317225 15274 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:22:45.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.317977 15272 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:22:45.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.318308 15272 heartbeater.cc:464] Registering TS with master...
05:22:45.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.319141 15272 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:22:45.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.320673 14741 ts_manager.cc:194] Registered new tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:22:45.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.322086 14894 ts_manager.cc:194] Registered new tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:22:45.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.322273 15273 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:22:45.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.322614 15273 heartbeater.cc:464] Registering TS with master...
05:22:45.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.323333 15273 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:22:45.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.324579 14894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.2:54999
05:22:45.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.325737 14816 ts_manager.cc:194] Registered new tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:22:45.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:45.665634 15282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:22:45.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:45.666240 15282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:22:45.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:45.666918 15282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:22:45.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:45.699944 15282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:22:45.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:45.700911 15282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.3
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.737787 15282 tablet_server_runner.cc:78] Tablet server non-default flags:
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.3:0
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.3
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:22:45.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.3
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/logs
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:22:45.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.739396 15282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:22:45.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:45.741187 15282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:22:45.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:45.762933 15291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:46.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.330385 15272 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:22:46.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:45.763002 15288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:46.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:45.763247 15289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:46.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:46.962720 15290 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
05:22:46.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.962831 15282 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
05:22:46.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.966766 15282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:22:46.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.969280 15282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:22:46.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.970665 15282 hybrid_clock.cc:648] HybridClock initialized: now 1783488166970629 us; error 43 us; skew 500 ppm
05:22:46.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.971621 15282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:22:46.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.978973 15282 webserver.cc:460] Webserver started at http://127.14.90.3:37591/ using document root <none> and password file <none>
05:22:46.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.980079 15282 fs_manager.cc:359] Metadata directory not provided
05:22:46.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.980341 15282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:22:46.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.980909 15282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:22:46.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.986256 15282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/instance:
05:22:46.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:22:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:46 on dist-test-slave-zq0g"
05:22:46.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.987603 15282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal/instance:
05:22:46.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:22:46.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:46 on dist-test-slave-zq0g"
05:22:46.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:46.995570 15282 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.007s	sys 0.001s
05:22:47.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.002224 15299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:22:47.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.003513 15282 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
05:22:47.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.003893 15282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:22:47.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:22:47.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:46 on dist-test-slave-zq0g"
05:22:47.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.004336 15282 fs_report.cc:383] FS layout report
05:22:47.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/data
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:22:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:22:47.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.055864 15282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:22:47.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.057570 15282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:22:47.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.058140 15282 kserver.cc:163] Server-wide thread pool size limit: 3276
05:22:47.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.061033 15282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:22:47.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.065668 15282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
05:22:47.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.065944 15282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
05:22:47.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.066305 15282 ts_tablet_manager.cc:610] Registered 0 tablets
05:22:47.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.066547 15282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
05:22:47.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.238261 15282 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.3:35001
05:22:47.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.238425 15411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.3:35001 every 8 connection(s)
05:22:47.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.241127 15282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:22:47.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.250111 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 15282
05:22:47.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.250727 14696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal/instance
05:22:47.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.308933 15414 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:22:47.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.309441 15414 heartbeater.cc:464] Registering TS with master...
05:22:47.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.310647 15412 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:22:47.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.310717 15414 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:22:47.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.311028 15412 heartbeater.cc:464] Registering TS with master...
05:22:47.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.311858 15412 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:22:47.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.312860 15413 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:22:47.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.313184 15413 heartbeater.cc:464] Registering TS with master...
05:22:47.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.313984 15413 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:22:47.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.315640 14741 ts_manager.cc:194] Registered new tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:22:47.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.316565 14816 ts_manager.cc:194] Registered new tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:22:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.318033 14894 ts_manager.cc:194] Registered new tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:22:47.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.320113 14894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.3:58669
05:22:47.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:47.325893 14696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
05:22:47.341 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
05:22:47.342 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

05:22:47.353 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.14.90.62"
      port: 44495
    }
    bound_http_address {
      host: "127.14.90.62"
      port: 46037
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.14.90.61"
      port: 34695
    }
    bound_http_address {
      host: "127.14.90.61"
      port: 40771
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.14.90.60"
      port: 44831
    }
    bound_http_address {
      host: "127.14.90.60"
      port: 36955
    }
  }
}

05:22:47.359 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

05:22:47.362 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.14.90.1"
      port: 46843
    }
    bound_http_address {
      host: "127.14.90.1"
      port: 34663
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.14.90.2"
      port: 45685
    }
    bound_http_address {
      host: "127.14.90.2"
      port: 34839
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.14.90.3"
      port: 35001
    }
    bound_http_address {
      host: "127.14.90.3"
      port: 37591
    }
  }
}

05:22:47.363 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
05:22:47.559 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
05:22:47.755 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.14.90.61:34695(127.14.90.61:34695)] connecting to peer
05:22:47.812 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.14.90.60:44831(127.14.90.60:44831)] connecting to peer
05:22:47.814 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.14.90.62:44495(127.14.90.62:44495)] connecting to peer
05:22:47.844 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.14.90.60:44831(127.14.90.60:44831)] Successfully connected to peer
05:22:47.847 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.14.90.61:34695(127.14.90.61:34695)] Successfully connected to peer
05:22:47.930 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.14.90.62:44495(127.14.90.62:44495)] Successfully connected to peer
05:22:47.963 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.14.90.60
05:22:47.963 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.14.90.61
05:22:48.172 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.14.90.62
05:22:48.208 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xf0eab43e, L:/127.0.0.1:47124 - R:/127.14.90.61:34695] using SASL/PLAIN
05:22:48.208 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x16fa1b3c, L:/127.0.0.1:59978 - R:/127.14.90.60:44831] using SASL/PLAIN
05:22:48.266 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x57079e00, L:/127.0.0.1:33902 - R:/127.14.90.62:44495] using SASL/PLAIN
05:22:48.276 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
05:22:48.278 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=568), Traces: [0ms] refreshing cache from master, [31ms] Sub RPC ConnectToMaster: sending RPC to server master-127.14.90.61:34695, [97ms] Sub RPC ConnectToMaster: sending RPC to server master-127.14.90.60:44831, [99ms] Sub RPC ConnectToMaster: sending RPC to server master-127.14.90.62:44495, [516ms] Sub RPC ConnectToMaster: received response from server master-127.14.90.60:44831: OK, [516ms] Sub RPC ConnectToMaster: received response from server master-127.14.90.61:34695: OK, [555ms] Sub RPC ConnectToMaster: received response from server master-127.14.90.62:44495: OK, deferred=null) after lookup
05:22:48.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.325259 15412 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:22:48.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.332711 14894 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59978:
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696"
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
05:22:48.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
05:22:48.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
05:22:48.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
05:22:48.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
05:22:48.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:22:48.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
05:22:48.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
05:22:48.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
05:22:48.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
05:22:48.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
05:22:48.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
05:22:48.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:22:48.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
05:22:48.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:48.337498 14894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
05:22:48.415 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 15f93422132b486e83f4973a897ac6c6
05:22:48.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.441445 15207 tablet_service.cc:1467] Processing CreateTablet for tablet 02161a9b84f144a4b9f0c67986f3fc46 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696 [id=15f93422132b486e83f4973a897ac6c6]), partition=RANGE (key) PARTITION UNBOUNDED
05:22:48.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.443928 15207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02161a9b84f144a4b9f0c67986f3fc46. 1 dirs total, 0 dirs full, 0 dirs failed
05:22:48.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.445525 15065 tablet_service.cc:1467] Processing CreateTablet for tablet 02161a9b84f144a4b9f0c67986f3fc46 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696 [id=15f93422132b486e83f4973a897ac6c6]), partition=RANGE (key) PARTITION UNBOUNDED
05:22:48.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.453084 15065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02161a9b84f144a4b9f0c67986f3fc46. 1 dirs total, 0 dirs full, 0 dirs failed
05:22:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.463202 15347 tablet_service.cc:1467] Processing CreateTablet for tablet 02161a9b84f144a4b9f0c67986f3fc46 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696 [id=15f93422132b486e83f4973a897ac6c6]), partition=RANGE (key) PARTITION UNBOUNDED
05:22:48.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.465447 15347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02161a9b84f144a4b9f0c67986f3fc46. 1 dirs total, 0 dirs full, 0 dirs failed
05:22:48.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.479916 15447 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Bootstrap starting.
05:22:48.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.486830 15447 tablet_bootstrap.cc:654] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Neither blocks nor log segments found. Creating new log.
05:22:48.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.489645 15448 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap starting.
05:22:48.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.490056 15447 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Log is configured to *not* fsync() on all Append() calls
05:22:48.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.500221 15448 tablet_bootstrap.cc:654] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Neither blocks nor log segments found. Creating new log.
05:22:48.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.502213 15447 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: No bootstrap required, opened a new log
05:22:48.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.502388 15448 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Log is configured to *not* fsync() on all Append() calls
05:22:48.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.502799 15447 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.007s
05:22:48.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.505309 15450 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Bootstrap starting.
05:22:48.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.509838 15448 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: No bootstrap required, opened a new log
05:22:48.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.510424 15448 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Time spent bootstrapping tablet: real 0.021s	user 0.011s	sys 0.004s
05:22:48.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.513404 15450 tablet_bootstrap.cc:654] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Neither blocks nor log segments found. Creating new log.
05:22:48.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.516223 15450 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Log is configured to *not* fsync() on all Append() calls
05:22:48.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.525250 15450 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: No bootstrap required, opened a new log
05:22:48.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.525853 15450 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Time spent bootstrapping tablet: real 0.021s	user 0.017s	sys 0.000s
05:22:48.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.531701 15447 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:48.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.534390 15447 raft_consensus.cc:374] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:22:48.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.534755 15447 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f24784906a4e2ebb7a4d8073e17074, State: Initialized, Role: FOLLOWER
05:22:48.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.535759 15447 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.540266 15447 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Time spent starting tablet: real 0.037s	user 0.025s	sys 0.006s
05:22:48.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:48.546000 15279 tablet.cc:2369] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Can't schedule compaction. Clean time has not been advanced past its initial value.
05:22:48.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.550834 15448 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:48.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.552011 15448 raft_consensus.cc:374] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:22:48.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.552363 15448 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969c70d1882541a4b320cca5f9979912, State: Initialized, Role: FOLLOWER
05:22:48.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.553346 15448 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:48.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.564210 15454 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:22:48.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.564800 15454 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:48.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.566625 15448 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Time spent starting tablet: real 0.056s	user 0.050s	sys 0.004s
05:22:48.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.566164 15450 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:48.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.567090 15450 raft_consensus.cc:374] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:22:48.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.567386 15450 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db66d29aaa8408c8acbad8bde8e4ae3, State: Initialized, Role: FOLLOWER
05:22:48.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.568302 15450 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:48.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.578161 15454 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:22:48.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.587724 15450 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Time spent starting tablet: real 0.062s	user 0.044s	sys 0.013s
05:22:48.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.614882 15453 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:22:48.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.615691 15453 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:48.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.618963 15453 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:22:48.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.639868 15367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "969c70d1882541a4b320cca5f9979912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" is_pre_election: true
05:22:48.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.640395 15085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969c70d1882541a4b320cca5f9979912" is_pre_election: true
05:22:48.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.641214 15085 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f24784906a4e2ebb7a4d8073e17074 in term 0.
05:22:48.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.641263 15367 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 969c70d1882541a4b320cca5f9979912 in term 0.
05:22:48.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.643805 15018 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, 969c70d1882541a4b320cca5f9979912; no voters: 
05:22:48.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.644695 15454 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 0 FOLLOWER]: Leader pre-election won for term 1
05:22:48.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.645044 15454 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
05:22:48.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.645340 15454 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 0 FOLLOWER]: Advancing to term 1
05:22:48.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.643487 15227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "969c70d1882541a4b320cca5f9979912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f24784906a4e2ebb7a4d8073e17074" is_pre_election: true
05:22:48.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.645880 15227 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 969c70d1882541a4b320cca5f9979912 in term 0.
05:22:48.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.644148 15160 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 969c70d1882541a4b320cca5f9979912, a2f24784906a4e2ebb7a4d8073e17074; no voters: 
05:22:48.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.647277 15453 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 0 FOLLOWER]: Leader pre-election won for term 1
05:22:48.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.647672 15453 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
05:22:48.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.648003 15453 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 0 FOLLOWER]: Advancing to term 1
05:22:48.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.653035 15454 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:48.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.655508 15454 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 1 election: Requested vote from peers a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:22:48.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.656562 15367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "969c70d1882541a4b320cca5f9979912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:22:48.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.657222 15367 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 0 FOLLOWER]: Advancing to term 1
05:22:48.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.657047 15453 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:48.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.658353 15227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "969c70d1882541a4b320cca5f9979912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f24784906a4e2ebb7a4d8073e17074"
05:22:48.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.658967 15453 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 1 election: Requested vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:22:48.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.659157 15227 raft_consensus.cc:2379] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 969c70d1882541a4b320cca5f9979912 in current term 1: Already voted for candidate a2f24784906a4e2ebb7a4d8073e17074 in this term.
05:22:48.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.661170 15085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969c70d1882541a4b320cca5f9979912"
05:22:48.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.662014 15085 raft_consensus.cc:2379] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2f24784906a4e2ebb7a4d8073e17074 in current term 1: Already voted for candidate 969c70d1882541a4b320cca5f9979912 in this term.
05:22:48.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.662679 15366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" is_pre_election: true
05:22:48.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.664405 15365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:22:48.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.665002 15367 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 969c70d1882541a4b320cca5f9979912 in term 1.
05:22:48.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.666169 15160 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2f24784906a4e2ebb7a4d8073e17074; no voters: 3db66d29aaa8408c8acbad8bde8e4ae3, 969c70d1882541a4b320cca5f9979912
05:22:48.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.666860 15453 raft_consensus.cc:2726] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
05:22:48.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.666595 15018 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, 969c70d1882541a4b320cca5f9979912; no voters: a2f24784906a4e2ebb7a4d8073e17074
05:22:48.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.667855 15454 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 1 FOLLOWER]: Leader election won for term 1
05:22:48.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.668294 15454 raft_consensus.cc:686] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 1 LEADER]: Becoming Leader. State: Replica: 969c70d1882541a4b320cca5f9979912, State: Running, Role: LEADER
05:22:48.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.669438 15454 consensus_queue.cc:237] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:48.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:48.683329 14892 catalog_manager.cc:5495] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 reported cstate change: term changed from 0 to 1, leader changed from <none> to 969c70d1882541a4b320cca5f9979912 (127.14.90.1). New cstate: current_term: 1 leader_uuid: "969c70d1882541a4b320cca5f9979912" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } health_report { overall_health: UNKNOWN } } }
05:22:48.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:48.741355 15137 tablet.cc:2369] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Can't schedule compaction. Clean time has not been advanced past its initial value.
05:22:48.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:48.783613 15419 tablet.cc:2369] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Can't schedule compaction. Clean time has not been advanced past its initial value.
05:22:48.940 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 15f93422132b486e83f4973a897ac6c6
05:22:48.961 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:48.968 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299993, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:22:48.969 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [11ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=null) after lookup
05:22:48.980 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:22:48.981 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] Successfully connected to peer
05:22:49.031 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xef215301, L:/127.0.0.1:34402 - R:/127.14.90.1:46843] using TOKEN/null
05:22:49.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:49.047049 15367 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 1 FOLLOWER]: Refusing update from remote peer 969c70d1882541a4b320cca5f9979912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
05:22:49.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:49.047093 15227 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 1 FOLLOWER]: Refusing update from remote peer 969c70d1882541a4b320cca5f9979912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
05:22:49.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:49.048792 15455 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
05:22:49.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:49.049537 15454 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
05:22:49.227 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:22:49.230 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.14.90.1:46843
05:22:49.233 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:22:49.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:49.233616 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 14976
05:22:49.281 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:22:49.282 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] handling channelInactive
05:22:49.283 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state READY due to: connection closed
05:22:49.284 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: connection closed
05:22:49.304 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:49.304 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:22:49.305 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2140980459(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@75b69a18 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6dcfa6ea -> wakeup thread Time-limited test)) after lookup
05:22:49.306 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:22:49.320 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.321 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.321 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.335 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:49.336 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:22:49.336 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=54), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [25ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [40ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [40ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [53ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2140980459(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@75b69a18 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6dcfa6ea -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733))) after lookup
05:22:49.338 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:22:49.339 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.339 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.339 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.375 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:49.375 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:22:49.376 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=94), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [25ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [40ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [40ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [53ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [56ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [58ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [58ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [89ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [93ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2140980459(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@75b69a18 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6dcfa6ea -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555))) after lookup
05:22:49.377 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:22:49.379 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.379 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.379 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.475 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:49.475 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:22:49.476 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=194), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [25ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [40ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [40ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [53ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [56ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [58ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [58ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [89ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [93ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [96ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [98ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [99ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [189ms] refreshing cache from master, [189ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [193ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2140980459(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@75b69a18 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555) -> (continuation of Deferred@1719959608 after retry RPC@169056533), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6dcfa6ea -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555) -> (continuation of Deferred@1719959608 after retry RPC@169056533))) after lookup
05:22:49.477 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:22:49.478 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.478 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.478 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.636 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:49.637 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:22:49.638 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=355), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [25ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [40ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [40ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [53ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [56ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [58ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [58ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [89ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [93ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [96ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [98ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [99ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [189ms] refreshing cache from master, [189ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [193ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [195ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [197ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [197ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [349ms] refreshing cache from master, [349ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [354ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2140980459(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@75b69a18 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555) -> (continuation of Deferred@1719959608 after retry RPC@169056533) -> (continuation of Deferred@1336567587 after retry RPC@109343858), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6dcfa6ea -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555) -> (continuation of Deferred@1719959608 after retry RPC@169056533) -> (continuation of Deferred@1336567587 after retry RPC@109343858))) after lookup
05:22:49.639 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:22:49.640 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.640 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:49.641 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:50.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.668846 15476 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 969c70d1882541a4b320cca5f9979912)
05:22:50.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.668471 15475 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 969c70d1882541a4b320cca5f9979912)
05:22:50.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.669262 15476 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:50.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.669387 15475 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:50.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.671558 15475 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:22:50.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.674578 15476 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:22:50.677 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:50.677 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:22:50.680 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1396), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [25ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [40ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [40ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [53ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [56ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [58ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [58ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [89ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [93ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [96ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [98ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [99ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [189ms] refreshing cache from master, [189ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [193ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [195ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [197ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [197ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [349ms] refreshing cache from master, [349ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [354ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [357ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [360ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [360ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1389ms] refreshing cache from master, [1389ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1394ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2140980459(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@75b69a18 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555) -> (continuation of Deferred@1719959608 after retry RPC@169056533) -> (continuation of Deferred@1336567587 after retry RPC@109343858) -> (continuation of Deferred@1661515222 after retry RPC@1175091745), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6dcfa6ea -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555) -> (continuation of Deferred@1719959608 after retry RPC@169056533) -> (continuation of Deferred@1336567587 after retry RPC@109343858) -> (continuation of Deferred@1661515222 after retry RPC@1175091745))) after lookup
05:22:50.681 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:22:50.682 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:50.682 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:50.683 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:50.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:50.686829 15300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:22:50.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:50.687589 15160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:22:50.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.687737 15367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" is_pre_election: true
05:22:50.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.688344 15367 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f24784906a4e2ebb7a4d8073e17074 in term 1.
05:22:50.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.690349 15160 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, a2f24784906a4e2ebb7a4d8073e17074; no voters: 
05:22:50.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.691335 15475 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 1 FOLLOWER]: Leader pre-election won for term 2
05:22:50.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.691689 15475 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 969c70d1882541a4b320cca5f9979912)
05:22:50.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.692016 15475 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 1 FOLLOWER]: Advancing to term 2
05:22:50.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:50.694749 15300 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:22:50.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:50.695046 15160 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:22:50.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.698266 15475 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:50.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.699472 15227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2f24784906a4e2ebb7a4d8073e17074" is_pre_election: true
05:22:50.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.700278 15227 raft_consensus.cc:2379] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3db66d29aaa8408c8acbad8bde8e4ae3 in current term 2: Already voted for candidate a2f24784906a4e2ebb7a4d8073e17074 in this term.
05:22:50.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.701589 15302 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3; no voters: 969c70d1882541a4b320cca5f9979912, a2f24784906a4e2ebb7a4d8073e17074
05:22:50.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.701946 15475 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 2 election: Requested vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:22:50.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.702862 15476 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 1 FOLLOWER]: Advancing to term 2
05:22:50.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.703663 15367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:22:50.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:50.707091 15160 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:22:50.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.708781 15476 raft_consensus.cc:2726] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
05:22:50.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.713068 15367 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f24784906a4e2ebb7a4d8073e17074 in term 2.
05:22:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.715723 15160 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, a2f24784906a4e2ebb7a4d8073e17074; no voters: 969c70d1882541a4b320cca5f9979912
05:22:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.716418 15475 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 2 FOLLOWER]: Leader election won for term 2
05:22:50.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.718133 15475 raft_consensus.cc:686] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 2 LEADER]: Becoming Leader. State: Replica: a2f24784906a4e2ebb7a4d8073e17074, State: Running, Role: LEADER
05:22:50.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.719485 15475 consensus_queue.cc:237] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:50.729269 14892 catalog_manager.cc:5495] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 reported cstate change: term changed from 1 to 2, leader changed from 969c70d1882541a4b320cca5f9979912 (127.14.90.1) to a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2). New cstate: current_term: 2 leader_uuid: "a2f24784906a4e2ebb7a4d8073e17074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } health_report { overall_health: UNKNOWN } } }
05:22:50.737 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:50.737 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:22:50.747 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1456), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [25ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [40ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [40ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [53ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [56ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [58ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [58ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [89ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [93ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [96ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [98ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [99ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [189ms] refreshing cache from master, [189ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [193ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [195ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [197ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [197ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [349ms] refreshing cache from master, [349ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [354ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [357ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [360ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [360ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1389ms] refreshing cache from master, [1389ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1394ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [1399ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [1402ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1402ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1449ms] refreshing cache from master, [1449ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1454ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2140980459(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@75b69a18 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555) -> (continuation of Deferred@1719959608 after retry RPC@169056533) -> (continuation of Deferred@1336567587 after retry RPC@109343858) -> (continuation of Deferred@1661515222 after retry RPC@1175091745) -> (continuation of Deferred@503531001 after retry RPC@1018869049), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6dcfa6ea -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555) -> (continuation of Deferred@1719959608 after retry RPC@169056533) -> (continuation of Deferred@1336567587 after retry RPC@109343858) -> (continuation of Deferred@1661515222 after retry RPC@1175091745) -> (continuation of Deferred@503531001 after retry RPC@1018869049))) after lookup
05:22:50.749 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:22:50.758 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:50.758 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:50.759 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:22:51.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:51.121075 15367 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 2 FOLLOWER]: Refusing update from remote peer a2f24784906a4e2ebb7a4d8073e17074: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
05:22:51.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:51.122388 15475 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
05:22:51.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:51.142539 15160 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 -> Peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Couldn't send request to peer 969c70d1882541a4b320cca5f9979912. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:22:53.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:53.782653 15160 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 -> Peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Couldn't send request to peer 969c70d1882541a4b320cca5f9979912. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
05:22:53.897 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:53.897 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:22:53.898 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=16, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=4616), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [25ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [40ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [40ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [53ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [56ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [58ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [58ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [89ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [93ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [96ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [98ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [99ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [189ms] refreshing cache from master, [189ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [193ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [195ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [197ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [197ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [349ms] refreshing cache from master, [349ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [354ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [357ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [360ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [360ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1389ms] refreshing cache from master, [1389ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1394ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [1399ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [1402ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1402ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1449ms] refreshing cache from master, [1449ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1454ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [1467ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [1477ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1478ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [4609ms] refreshing cache from master, [4610ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [4614ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2140980459(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@75b69a18 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555) -> (continuation of Deferred@1719959608 after retry RPC@169056533) -> (continuation of Deferred@1336567587 after retry RPC@109343858) -> (continuation of Deferred@1661515222 after retry RPC@1175091745) -> (continuation of Deferred@503531001 after retry RPC@1018869049) -> (continuation of Deferred@1486064511 after retry RPC@829029929), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6dcfa6ea -> wakeup thread Time-limited test -> (continuation of Deferred@1260962465 after retry RPC@1210366733) -> (continuation of Deferred@1842287089 after retry RPC@391079555) -> (continuation of Deferred@1719959608 after retry RPC@169056533) -> (continuation of Deferred@1336567587 after retry RPC@109343858) -> (continuation of Deferred@1661515222 after retry RPC@1175091745) -> (continuation of Deferred@503531001 after retry RPC@1018869049) -> (continuation of Deferred@1486064511 after retry RPC@829029929))) after lookup
05:22:53.899 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:22:53.900 [DEBUG - kudu-nio-1] (Connection.java:757) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] Successfully connected to peer
05:22:53.932 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xcbf0754b, L:/127.0.0.1:41336 - R:/127.14.90.2:45685] using TOKEN/null
05:22:53.985 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.14.90.1:46843
05:22:53.987 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:22:53.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:53.989380 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:53.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:53.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/logs
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.1:46843
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.1
05:22:53.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:53.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34663
05:22:53.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.1
05:22:53.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:53.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:53.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:53.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:22:54.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:54.333159 15519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:22:54.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:54.333659 15519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:22:54.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:54.334115 15519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:22:54.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:54.367944 15519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:22:54.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:54.368927 15519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.1
05:22:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:54.405292 15519 tablet_server_runner.cc:78] Tablet server non-default flags:
05:22:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.1:46843
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.1
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34663
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.1
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/logs
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:22:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:22:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:22:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:22:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:22:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:22:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:22:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:22:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:22:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:54.407033 15519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:22:54.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:54.408761 15519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:22:55.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:54.431574 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:55.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:54.434000 15519 server_base.cc:1034] running on GCE node
05:22:55.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:54.431540 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:55.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:54.432552 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:22:55.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.673295 15519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:22:55.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.675707 15519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:22:55.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.677146 15519 hybrid_clock.cc:648] HybridClock initialized: now 1783488175677111 us; error 70 us; skew 500 ppm
05:22:55.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.678000 15519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:22:55.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.691601 15519 webserver.cc:460] Webserver started at http://127.14.90.1:34663/ using document root <none> and password file <none>
05:22:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.692662 15519 fs_manager.cc:359] Metadata directory not provided
05:22:55.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.692936 15519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:22:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.702412 15519 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.006s	sys 0.001s
05:22:55.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.707597 15543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:22:55.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.708841 15519 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.003s
05:22:55.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.709208 15519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:22:55.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "969c70d1882541a4b320cca5f9979912"
05:22:55.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:42 on dist-test-slave-zq0g"
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.711696 15519 fs_report.cc:383] FS layout report
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/data
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:22:55.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:22:55.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.750263 15519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:22:55.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.752252 15519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:22:55.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.752815 15519 kserver.cc:163] Server-wide thread pool size limit: 3276
05:22:55.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.755671 15519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:22:55.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.762080 15550 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:22:55.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.771597 15519 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:22:55.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.771893 15519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.001s
05:22:55.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.772248 15519 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:22:55.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.777140 15519 ts_tablet_manager.cc:610] Registered 1 tablets
05:22:55.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.777364 15519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.003s	sys 0.000s
05:22:55.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.777805 15550 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap starting.
05:22:55.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.844952 15550 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Log is configured to *not* fsync() on all Append() calls
05:22:55.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.874979 15550 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:22:55.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.875888 15550 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap complete.
05:22:55.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.877630 15550 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Time spent bootstrapping tablet: real 0.100s	user 0.055s	sys 0.038s
05:22:55.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.899945 15550 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:55.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.903313 15550 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969c70d1882541a4b320cca5f9979912, State: Initialized, Role: FOLLOWER
05:22:55.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.904570 15550 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:55.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.909840 15550 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Time spent starting tablet: real 0.032s	user 0.025s	sys 0.004s
05:22:55.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:55.927323 15160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111) [suppressed 10 similar messages]
05:22:55.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.971344 15519 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.1:46843
05:22:55.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.971524 15661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.1:46843 every 8 connection(s)
05:22:55.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.973837 15519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:22:55.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:55.975519 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 15519
05:22:55.976 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:22:56.014 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:22:56.016 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.14.90.2:45685
05:22:56.016 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:22:56.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.016765 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 15140
05:22:56.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.026746 15662 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:22:56.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.027323 15662 heartbeater.cc:464] Registering TS with master...
05:22:56.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.028823 15663 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:22:56.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.029217 15663 heartbeater.cc:464] Registering TS with master...
05:22:56.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.030200 15662 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:22:56.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.030597 15663 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:22:56.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.035305 14892 ts_manager.cc:194] Re-registered known tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:22:56.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.036137 14816 ts_manager.cc:194] Re-registered known tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:22:56.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.039105 14892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.1:50295
05:22:56.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.046512 15664 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:22:56.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.046942 15664 heartbeater.cc:464] Registering TS with master...
05:22:56.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.047726 15664 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:22:56.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:56.051738 14741 ts_manager.cc:194] Re-registered known tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:22:56.053 [DEBUG - kudu-nio-1] (Connection.java:239) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] handling channelInactive
05:22:56.053 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state READY due to: connection closed
05:22:56.054 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:22:56.055 [DEBUG - Time-limited test] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:22:56.057 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.057 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.057 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.076 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:56.076 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:22:56.077 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [20ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@19622839(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@1bb965b5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57f382f5 -> wakeup thread Time-limited test)) after lookup
05:22:56.078 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:22:56.079 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.079 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.079 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.095 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:56.096 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:22:56.098 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [20ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [22ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [24ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [24ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [39ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@19622839(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@1bb965b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57f382f5 -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801))) after lookup
05:22:56.099 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:22:56.102 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.102 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.103 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.136 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:56.136 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:22:56.137 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=82), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [20ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [22ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [24ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [24ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [39ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [43ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [48ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [48ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [80ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@19622839(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@1bb965b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801) -> (continuation of Deferred@1659493921 after retry RPC@1598445805), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57f382f5 -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801) -> (continuation of Deferred@1659493921 after retry RPC@1598445805))) after lookup
05:22:56.138 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:22:56.140 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.140 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.141 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.216 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:56.216 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:22:56.217 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=161), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [20ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [22ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [24ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [24ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [39ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [43ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [48ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [48ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [80ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [83ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [85ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [86ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [155ms] refreshing cache from master, [155ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [159ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@19622839(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@1bb965b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801) -> (continuation of Deferred@1659493921 after retry RPC@1598445805) -> (continuation of Deferred@541178798 after retry RPC@2055792793), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57f382f5 -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801) -> (continuation of Deferred@1659493921 after retry RPC@1598445805) -> (continuation of Deferred@541178798 after retry RPC@2055792793))) after lookup
05:22:56.218 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:22:56.220 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.220 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.221 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.295 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:56.295 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:22:56.296 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=241), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [20ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [22ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [24ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [24ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [39ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [43ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [48ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [48ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [80ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [83ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [85ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [86ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [155ms] refreshing cache from master, [155ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [159ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [163ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [166ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [166ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [235ms] refreshing cache from master, [235ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [239ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@19622839(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@1bb965b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801) -> (continuation of Deferred@1659493921 after retry RPC@1598445805) -> (continuation of Deferred@541178798 after retry RPC@2055792793) -> (continuation of Deferred@1340194876 after retry RPC@1913664889), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57f382f5 -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801) -> (continuation of Deferred@1659493921 after retry RPC@1598445805) -> (continuation of Deferred@541178798 after retry RPC@2055792793) -> (continuation of Deferred@1340194876 after retry RPC@1913664889))) after lookup
05:22:56.297 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:22:56.298 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.298 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.298 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.896 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:56.896 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:22:56.897 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=841), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [20ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [22ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [24ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [24ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [39ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [43ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [48ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [48ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [80ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [83ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [85ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [86ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [155ms] refreshing cache from master, [155ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [159ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [163ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [166ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [166ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [235ms] refreshing cache from master, [235ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [239ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [242ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [243ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [243ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [836ms] refreshing cache from master, [836ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [840ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@19622839(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@1bb965b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801) -> (continuation of Deferred@1659493921 after retry RPC@1598445805) -> (continuation of Deferred@541178798 after retry RPC@2055792793) -> (continuation of Deferred@1340194876 after retry RPC@1913664889) -> (continuation of Deferred@1826259730 after retry RPC@350676227), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57f382f5 -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801) -> (continuation of Deferred@1659493921 after retry RPC@1598445805) -> (continuation of Deferred@541178798 after retry RPC@2055792793) -> (continuation of Deferred@1340194876 after retry RPC@1913664889) -> (continuation of Deferred@1826259730 after retry RPC@350676227))) after lookup
05:22:56.898 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:22:56.900 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.900 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:56.900 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:22:57.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.044041 15662 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:22:57.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.111537 15673 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:22:57.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.111980 15673 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:57.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.114339 15673 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:22:57.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:57.118145 15546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111)
05:22:57.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:57.124039 15546 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111)
05:22:57.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.129037 15367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "969c70d1882541a4b320cca5f9979912" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" is_pre_election: true
05:22:57.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.130939 15544 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 969c70d1882541a4b320cca5f9979912; no voters: 3db66d29aaa8408c8acbad8bde8e4ae3, a2f24784906a4e2ebb7a4d8073e17074
05:22:57.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.131706 15673 raft_consensus.cc:2726] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
05:22:57.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.209432 15678 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a2f24784906a4e2ebb7a4d8073e17074)
05:22:57.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.209901 15678 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:57.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.211679 15678 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:22:57.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:57.214977 15302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111)
05:22:57.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:57.220589 15302 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111)
05:22:57.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.229558 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "969c70d1882541a4b320cca5f9979912" is_pre_election: true
05:22:57.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.230461 15613 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db66d29aaa8408c8acbad8bde8e4ae3 in term 1.
05:22:57.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.232139 15300 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, 969c70d1882541a4b320cca5f9979912; no voters: a2f24784906a4e2ebb7a4d8073e17074
05:22:57.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.233125 15678 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 2 FOLLOWER]: Leader pre-election won for term 3
05:22:57.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.233503 15678 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a2f24784906a4e2ebb7a4d8073e17074)
05:22:57.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.233870 15678 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 2 FOLLOWER]: Advancing to term 3
05:22:57.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.240820 15678 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:57.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.242765 15678 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 3 election: Requested vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:22:57.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.243665 15613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "969c70d1882541a4b320cca5f9979912"
05:22:57.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.244269 15613 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 1 FOLLOWER]: Advancing to term 3
05:22:57.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:57.248291 15302 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111)
05:22:57.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.257571 15613 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db66d29aaa8408c8acbad8bde8e4ae3 in term 3.
05:22:57.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.258971 15300 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, 969c70d1882541a4b320cca5f9979912; no voters: a2f24784906a4e2ebb7a4d8073e17074
05:22:57.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.259598 15678 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 3 FOLLOWER]: Leader election won for term 3
05:22:57.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.260849 15678 raft_consensus.cc:686] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 3 LEADER]: Becoming Leader. State: Replica: 3db66d29aaa8408c8acbad8bde8e4ae3, State: Running, Role: LEADER
05:22:57.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.261869 15678 consensus_queue.cc:237] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:22:57.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.274096 14892 catalog_manager.cc:5495] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 reported cstate change: term changed from 2 to 3, leader changed from a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2) to 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3). New cstate: current_term: 3 leader_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } health_report { overall_health: HEALTHY } } }
05:22:57.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.708189 15613 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 3 FOLLOWER]: Refusing update from remote peer 3db66d29aaa8408c8acbad8bde8e4ae3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
05:22:57.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:57.709703 15678 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
05:22:57.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:57.710850 15302 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 -> Peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Couldn't send request to peer a2f24784906a4e2ebb7a4d8073e17074. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:22:59.075 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:22:59.075 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:22:59.076 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=3020), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [20ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [22ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [24ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [24ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [39ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [43ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [48ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [48ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [80ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [83ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [85ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [86ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [155ms] refreshing cache from master, [155ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [159ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [163ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [166ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [166ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [235ms] refreshing cache from master, [235ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [239ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [242ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [243ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [243ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [836ms] refreshing cache from master, [836ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [840ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [843ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [845ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [846ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [3015ms] refreshing cache from master, [3015ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [3019ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@19622839(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@1bb965b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801) -> (continuation of Deferred@1659493921 after retry RPC@1598445805) -> (continuation of Deferred@541178798 after retry RPC@2055792793) -> (continuation of Deferred@1340194876 after retry RPC@1913664889) -> (continuation of Deferred@1826259730 after retry RPC@350676227) -> (continuation of Deferred@488676314 after retry RPC@1053670272), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57f382f5 -> wakeup thread Time-limited test -> (continuation of Deferred@92219774 after retry RPC@731041801) -> (continuation of Deferred@1659493921 after retry RPC@1598445805) -> (continuation of Deferred@541178798 after retry RPC@2055792793) -> (continuation of Deferred@1340194876 after retry RPC@1913664889) -> (continuation of Deferred@1826259730 after retry RPC@350676227) -> (continuation of Deferred@488676314 after retry RPC@1053670272))) after lookup
05:22:59.077 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:22:59.078 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] Successfully connected to peer
05:22:59.103 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x67dce721, L:/127.0.0.1:53466 - R:/127.14.90.3:35001] using TOKEN/null
05:22:59.159 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.14.90.2:45685
05:22:59.160 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:59.161887 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/logs
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.2:45685
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.2
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34839
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.2
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:59.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:22:59.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:59.493963 15715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:22:59.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:59.494449 15715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:22:59.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:59.494987 15715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:22:59.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:59.529562 15715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:22:59.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:59.530586 15715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.2
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:59.569049 15715 tablet_server_runner.cc:78] Tablet server non-default flags:
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.2:45685
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:22:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.2
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34839
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.2
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/logs
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:22:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:22:59.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:22:59.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:22:59.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:22:59.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:22:59.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:59.570348 15715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:22:59.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:59.572023 15715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:22:59.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:59.591113 15724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:00.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:00.273378 15302 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 -> Peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Couldn't send request to peer a2f24784906a4e2ebb7a4d8073e17074. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
05:23:00.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:22:59.592783 15715 server_base.cc:1034] running on GCE node
05:23:00.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:59.591217 15723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:00.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:22:59.591194 15726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:00.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.830235 15715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:23:00.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.833055 15715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:23:00.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.834591 15715 hybrid_clock.cc:648] HybridClock initialized: now 1783488180834506 us; error 82 us; skew 500 ppm
05:23:00.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.835599 15715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:23:00.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.848533 15715 webserver.cc:460] Webserver started at http://127.14.90.2:34839/ using document root <none> and password file <none>
05:23:00.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.849606 15715 fs_manager.cc:359] Metadata directory not provided
05:23:00.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.849881 15715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:23:00.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.859143 15715 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.007s	sys 0.000s
05:23:00.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.864122 15735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:23:00.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.865314 15715 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.004s
05:23:00.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.865717 15715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:00.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a2f24784906a4e2ebb7a4d8073e17074"
05:23:00.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:45 on dist-test-slave-zq0g"
05:23:00.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.868333 15715 fs_report.cc:383] FS layout report
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/data
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:23:00.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:23:00.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:23:00.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:23:00.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.919414 15715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:23:00.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.921026 15715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:23:00.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.921574 15715 kserver.cc:163] Server-wide thread pool size limit: 3276
05:23:00.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.924808 15715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:23:00.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.930980 15742 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:23:00.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.938898 15715 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:23:00.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.939251 15715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
05:23:00.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.939886 15715 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:23:00.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.945369 15715 ts_tablet_manager.cc:610] Registered 1 tablets
05:23:00.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.945613 15715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.004s	sys 0.000s
05:23:00.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:00.946576 15742 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Bootstrap starting.
05:23:01.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.009320 15742 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Log is configured to *not* fsync() on all Append() calls
05:23:01.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.058775 15742 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:23:01.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.060068 15742 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Bootstrap complete.
05:23:01.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.061638 15742 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Time spent bootstrapping tablet: real 0.115s	user 0.068s	sys 0.043s
05:23:01.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.086668 15742 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:01.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.089478 15742 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f24784906a4e2ebb7a4d8073e17074, State: Initialized, Role: FOLLOWER
05:23:01.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.091070 15742 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:01.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.096186 15742 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Time spent starting tablet: real 0.034s	user 0.030s	sys 0.006s
05:23:01.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.137686 15715 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.2:45685
05:23:01.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.137874 15850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.2:45685 every 8 connection(s)
05:23:01.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.140616 15715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:23:01.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.146296 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 15715
05:23:01.147 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:01.176 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:23:01.177 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.14.90.3:35001
05:23:01.177 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:23:01.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.178145 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 15282
05:23:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.207755 15851 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:23:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.208261 15851 heartbeater.cc:464] Registering TS with master...
05:23:01.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.209548 15851 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:23:01.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.210041 15852 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:23:01.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.210386 15852 heartbeater.cc:464] Registering TS with master...
05:23:01.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.211263 15852 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:23:01.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.212373 15853 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:23:01.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.212719 15853 heartbeater.cc:464] Registering TS with master...
05:23:01.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.213522 15853 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:23:01.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.214573 14892 ts_manager.cc:194] Re-registered known tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:01.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.217104 14816 ts_manager.cc:194] Re-registered known tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:01.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.218369 14892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.2:33765
05:23:01.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:01.220052 14740 ts_manager.cc:194] Re-registered known tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:01.225 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] handling channelInactive
05:23:01.225 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state READY due to: connection closed
05:23:01.225 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:01.226 [DEBUG - Time-limited test] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:01.229 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.229 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.230 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.255 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:01.255 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:01.255 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [4ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1156683741(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@217c71e3 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@d81a042 -> wakeup thread Time-limited test)) after lookup
05:23:01.256 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:01.257 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.257 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.257 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.275 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:01.275 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:01.276 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=50), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [4ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [30ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [31ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [31ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [49ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1156683741(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@217c71e3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@67434331 after retry RPC@181822659), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@d81a042 -> wakeup thread Time-limited test -> (continuation of Deferred@67434331 after retry RPC@181822659))) after lookup
05:23:01.277 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:01.279 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.280 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.280 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.295 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:01.295 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:01.296 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=69), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [4ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [30ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [31ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [31ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [49ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [51ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [54ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [54ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [68ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1156683741(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@217c71e3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@67434331 after retry RPC@181822659) -> (continuation of Deferred@808805438 after retry RPC@159186567), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@d81a042 -> wakeup thread Time-limited test -> (continuation of Deferred@67434331 after retry RPC@181822659) -> (continuation of Deferred@808805438 after retry RPC@159186567))) after lookup
05:23:01.296 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:01.297 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.297 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.298 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.355 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:01.356 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:01.357 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=130), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [4ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [30ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [31ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [31ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [49ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [51ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [54ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [54ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [68ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [70ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [71ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [72ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [129ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1156683741(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@217c71e3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@67434331 after retry RPC@181822659) -> (continuation of Deferred@808805438 after retry RPC@159186567) -> (continuation of Deferred@1093423141 after retry RPC@708780706), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@d81a042 -> wakeup thread Time-limited test -> (continuation of Deferred@67434331 after retry RPC@181822659) -> (continuation of Deferred@808805438 after retry RPC@159186567) -> (continuation of Deferred@1093423141 after retry RPC@708780706))) after lookup
05:23:01.358 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:01.359 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.359 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.360 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.835 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:01.835 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:01.835 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=609), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [4ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [30ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [31ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [31ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [49ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [51ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [54ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [54ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [68ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [70ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [71ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [72ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [129ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [131ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [134ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [134ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [604ms] refreshing cache from master, [604ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [608ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1156683741(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@217c71e3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@67434331 after retry RPC@181822659) -> (continuation of Deferred@808805438 after retry RPC@159186567) -> (continuation of Deferred@1093423141 after retry RPC@708780706) -> (continuation of Deferred@1439035841 after retry RPC@113042409), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@d81a042 -> wakeup thread Time-limited test -> (continuation of Deferred@67434331 after retry RPC@181822659) -> (continuation of Deferred@808805438 after retry RPC@159186567) -> (continuation of Deferred@1093423141 after retry RPC@708780706) -> (continuation of Deferred@1439035841 after retry RPC@113042409))) after lookup
05:23:01.836 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:01.837 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.837 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:01.838 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:02.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.150815 15862 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3db66d29aaa8408c8acbad8bde8e4ae3)
05:23:02.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.151304 15862 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:02.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.158996 15544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111)
05:23:02.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.159209 15862 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:02.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.162663 15544 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001): Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111)
05:23:02.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.173059 15805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "969c70d1882541a4b320cca5f9979912" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "a2f24784906a4e2ebb7a4d8073e17074" is_pre_election: true
05:23:02.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.173754 15805 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 969c70d1882541a4b320cca5f9979912 in term 2.
05:23:02.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.174907 15546 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 969c70d1882541a4b320cca5f9979912, a2f24784906a4e2ebb7a4d8073e17074; no voters: 3db66d29aaa8408c8acbad8bde8e4ae3
05:23:02.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.175525 15862 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 3 FOLLOWER]: Leader pre-election won for term 4
05:23:02.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.175853 15862 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3db66d29aaa8408c8acbad8bde8e4ae3)
05:23:02.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.176072 15862 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 3 FOLLOWER]: Advancing to term 4
05:23:02.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.180263 15862 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:02.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.181780 15862 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 4 election: Requested vote from peers a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:02.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.182682 15805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "969c70d1882541a4b320cca5f9979912" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "a2f24784906a4e2ebb7a4d8073e17074"
05:23:02.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.183147 15805 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 2 FOLLOWER]: Advancing to term 4
05:23:02.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.185207 15544 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001): Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111)
05:23:02.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.189924 15805 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 969c70d1882541a4b320cca5f9979912 in term 4.
05:23:02.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.191177 15546 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 969c70d1882541a4b320cca5f9979912, a2f24784906a4e2ebb7a4d8073e17074; no voters: 3db66d29aaa8408c8acbad8bde8e4ae3
05:23:02.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.191773 15862 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 4 FOLLOWER]: Leader election won for term 4
05:23:02.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.192961 15862 raft_consensus.cc:686] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 4 LEADER]: Becoming Leader. State: Replica: 969c70d1882541a4b320cca5f9979912, State: Running, Role: LEADER
05:23:02.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.193817 15862 consensus_queue.cc:237] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:02.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.201835 14892 catalog_manager.cc:5495] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 reported cstate change: term changed from 3 to 4, leader changed from 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3) to 969c70d1882541a4b320cca5f9979912 (127.14.90.1). New cstate: current_term: 4 leader_uuid: "969c70d1882541a4b320cca5f9979912" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } health_report { overall_health: UNKNOWN } } }
05:23:02.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.223214 15851 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:23:02.374 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:02.375 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:02.375 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1149), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [4ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [30ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [31ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [31ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [49ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [51ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [54ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [54ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [68ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [70ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [71ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [72ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [129ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [131ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [134ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [134ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [604ms] refreshing cache from master, [604ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [608ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [610ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [611ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [612ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [1144ms] refreshing cache from master, [1144ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1148ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1156683741(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@217c71e3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@67434331 after retry RPC@181822659) -> (continuation of Deferred@808805438 after retry RPC@159186567) -> (continuation of Deferred@1093423141 after retry RPC@708780706) -> (continuation of Deferred@1439035841 after retry RPC@113042409) -> (continuation of Deferred@1850221147 after retry RPC@957061791), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@d81a042 -> wakeup thread Time-limited test -> (continuation of Deferred@67434331 after retry RPC@181822659) -> (continuation of Deferred@808805438 after retry RPC@159186567) -> (continuation of Deferred@1093423141 after retry RPC@708780706) -> (continuation of Deferred@1439035841 after retry RPC@113042409) -> (continuation of Deferred@1850221147 after retry RPC@957061791))) after lookup
05:23:02.376 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:02.378 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] Successfully connected to peer
05:23:02.403 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x6864806c, L:/127.0.0.1:39508 - R:/127.14.90.1:46843] using TOKEN/null
05:23:02.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.423970 15805 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 4 FOLLOWER]: Refusing update from remote peer 969c70d1882541a4b320cca5f9979912: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
05:23:02.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.425153 15544 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 -> Peer 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001): Couldn't send request to peer 3db66d29aaa8408c8acbad8bde8e4ae3. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:23:02.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.425793 15862 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
05:23:02.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.446997 15883 mvcc.cc:205] Tried to move back new op lower bound from 7305167581628313600 to 7305167574073307136. Current Snapshot: MvccSnapshot[applied={T|T < 7305167581628313600}]
05:23:02.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.459580 15881 mvcc.cc:205] Tried to move back new op lower bound from 7305167595185451008 to 7305167594274418688. Current Snapshot: MvccSnapshot[applied={T|T < 7305167595185451008}]
05:23:02.507 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.14.90.3:35001
05:23:02.507 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.508951 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/logs
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.3:35001
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.3
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37591
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.3
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:02.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:02.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:02.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:23:02.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.846763 15890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:23:02.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.847241 15890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:23:02.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.847694 15890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:23:02.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.880558 15890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:23:02.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.881383 15890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.3
05:23:02.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.916210 15890 tablet_server_runner.cc:78] Tablet server non-default flags:
05:23:02.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:02.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:02.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:02.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:23:02.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.3:35001
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.3
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37591
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.3
05:23:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/logs
05:23:02.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:02.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:23:02.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:23:02.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:23:02.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:23:02.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:23:02.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:23:02.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:23:02.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:23:02.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:23:02.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.917492 15890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:23:02.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.919222 15890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:23:02.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.940171 15899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.940160 15897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:02.940176 15896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:04.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:02.942160 15890 server_base.cc:1034] running on GCE node
05:23:04.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.170922 15890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:23:04.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.173135 15890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:23:04.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.174552 15890 hybrid_clock.cc:648] HybridClock initialized: now 1783488184174457 us; error 59 us; skew 500 ppm
05:23:04.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.175336 15890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:23:04.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.181831 15890 webserver.cc:460] Webserver started at http://127.14.90.3:37591/ using document root <none> and password file <none>
05:23:04.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.182857 15890 fs_manager.cc:359] Metadata directory not provided
05:23:04.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.183070 15890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:23:04.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.193746 15890 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.000s
05:23:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.199313 15914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:23:04.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.200518 15890 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.004s
05:23:04.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.200922 15890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:04.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:23:04.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:46 on dist-test-slave-zq0g"
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.203563 15890 fs_report.cc:383] FS layout report
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/data
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:23:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:23:04.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.254712 15890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:23:04.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.256752 15890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:23:04.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.257347 15890 kserver.cc:163] Server-wide thread pool size limit: 3276
05:23:04.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.260804 15890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:23:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.267665 15921 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:23:04.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.275758 15890 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:23:04.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.276016 15890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
05:23:04.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.276245 15890 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:23:04.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.281208 15890 ts_tablet_manager.cc:610] Registered 1 tablets
05:23:04.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.281407 15890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.002s	sys 0.001s
05:23:04.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.282025 15921 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Bootstrap starting.
05:23:04.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.375527 15921 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Log is configured to *not* fsync() on all Append() calls
05:23:04.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.476202 15921 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:23:04.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.477782 15921 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Bootstrap complete.
05:23:04.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.480345 15921 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Time spent bootstrapping tablet: real 0.199s	user 0.109s	sys 0.040s
05:23:04.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.516420 15921 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:04.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.520995 15921 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db66d29aaa8408c8acbad8bde8e4ae3, State: Initialized, Role: FOLLOWER
05:23:04.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.522537 15921 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:04.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.528191 15921 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Time spent starting tablet: real 0.047s	user 0.029s	sys 0.012s
05:23:04.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.564958 15890 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.3:35001
05:23:04.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.565155 16030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.3:35001 every 8 connection(s)
05:23:04.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.567500 15890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:23:04.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.569844 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 15890
05:23:04.571 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:04.608 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:23:04.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.620863 16032 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:23:04.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.624761 16032 heartbeater.cc:464] Registering TS with master...
05:23:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.625847 16031 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:23:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.626221 16031 heartbeater.cc:464] Registering TS with master...
05:23:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.626212 16032 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:23:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.627106 16031 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:23:04.637 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.14.90.1:46843
05:23:04.637 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:23:04.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.631318 14816 ts_manager.cc:194] Re-registered known tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:04.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.634116 14894 ts_manager.cc:194] Re-registered known tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:04.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.637962 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 15519
05:23:04.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.637669 14894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.3:35175
05:23:04.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.647668 16033 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:23:04.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.648082 16033 heartbeater.cc:464] Registering TS with master...
05:23:04.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.648813 16033 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:23:04.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:04.653298 14740 ts_manager.cc:194] Re-registered known tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:04.677 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] handling channelInactive
05:23:04.678 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state READY due to: connection closed
05:23:04.679 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:04.680 [DEBUG - Time-limited test] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:04.682 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.682 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.682 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.695 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:04.695 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:04.696 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=16), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [3ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [15ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@705369251(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2fa37815 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a893713 -> wakeup thread Time-limited test)) after lookup
05:23:04.696 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:04.699 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.699 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.699 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.715 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:04.715 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:04.715 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [3ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [15ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [17ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [20ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [21ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [35ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@705369251(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2fa37815 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@993419682 after retry RPC@1758787704), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a893713 -> wakeup thread Time-limited test -> (continuation of Deferred@993419682 after retry RPC@1758787704))) after lookup
05:23:04.716 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:04.717 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.717 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.717 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.757 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:04.757 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:04.758 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=79), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [3ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [15ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [17ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [20ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [21ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [35ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [37ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [38ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [38ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [77ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@705369251(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2fa37815 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@993419682 after retry RPC@1758787704) -> (continuation of Deferred@476314037 after retry RPC@1523318892), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a893713 -> wakeup thread Time-limited test -> (continuation of Deferred@993419682 after retry RPC@1758787704) -> (continuation of Deferred@476314037 after retry RPC@1523318892))) after lookup
05:23:04.759 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:04.762 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.762 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.763 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.814 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:04.814 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:04.815 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=135), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [3ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [15ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [17ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [20ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [21ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [35ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [37ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [38ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [38ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [77ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [80ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [83ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [84ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [131ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [135ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@705369251(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2fa37815 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@993419682 after retry RPC@1758787704) -> (continuation of Deferred@476314037 after retry RPC@1523318892) -> (continuation of Deferred@1471830855 after retry RPC@1833959100), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a893713 -> wakeup thread Time-limited test -> (continuation of Deferred@993419682 after retry RPC@1758787704) -> (continuation of Deferred@476314037 after retry RPC@1523318892) -> (continuation of Deferred@1471830855 after retry RPC@1833959100))) after lookup
05:23:04.816 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:04.816 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.817 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:04.817 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:05.114 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:05.115 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:05.115 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=435), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [3ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [15ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [17ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [20ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [21ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [35ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [37ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [38ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [38ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [77ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [80ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [83ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [84ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [131ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [135ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [136ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [138ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [138ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [435ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@705369251(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2fa37815 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@993419682 after retry RPC@1758787704) -> (continuation of Deferred@476314037 after retry RPC@1523318892) -> (continuation of Deferred@1471830855 after retry RPC@1833959100) -> (continuation of Deferred@1628293648 after retry RPC@1420071031), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a893713 -> wakeup thread Time-limited test -> (continuation of Deferred@993419682 after retry RPC@1758787704) -> (continuation of Deferred@476314037 after retry RPC@1523318892) -> (continuation of Deferred@1471830855 after retry RPC@1833959100) -> (continuation of Deferred@1628293648 after retry RPC@1420071031))) after lookup
05:23:05.116 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:05.118 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:05.118 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:05.118 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:05.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.656754 16031 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:23:05.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.809468 16042 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:23:05.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.809916 16042 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:05.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.812397 16042 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:05.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:05.817282 15915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:23:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:05.821378 15915 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:23:05.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.826296 15805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "a2f24784906a4e2ebb7a4d8073e17074" is_pre_election: true
05:23:05.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.827862 15917 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3; no voters: 969c70d1882541a4b320cca5f9979912, a2f24784906a4e2ebb7a4d8073e17074
05:23:05.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.828802 16042 raft_consensus.cc:2726] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
05:23:05.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.838207 16047 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 969c70d1882541a4b320cca5f9979912)
05:23:05.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.838900 16047 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:05.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.841665 16047 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:05.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:05.844966 15736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:23:05.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:05.849241 15736 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:23:05.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.857015 15985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" is_pre_election: true
05:23:05.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.857782 15985 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f24784906a4e2ebb7a4d8073e17074 in term 3.
05:23:05.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.859174 15736 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, a2f24784906a4e2ebb7a4d8073e17074; no voters: 969c70d1882541a4b320cca5f9979912
05:23:05.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.859972 16047 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 4 FOLLOWER]: Leader pre-election won for term 5
05:23:05.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.860255 16047 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 969c70d1882541a4b320cca5f9979912)
05:23:05.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.860494 16047 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 4 FOLLOWER]: Advancing to term 5
05:23:05.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.864662 16047 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:05.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.865994 16047 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 5 election: Requested vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:05.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.867774 15985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:23:05.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.868397 15985 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 3 FOLLOWER]: Advancing to term 5
05:23:05.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:05.869287 15736 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:23:05.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.875342 15985 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f24784906a4e2ebb7a4d8073e17074 in term 5.
05:23:05.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.876389 15736 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, a2f24784906a4e2ebb7a4d8073e17074; no voters: 969c70d1882541a4b320cca5f9979912
05:23:05.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.876999 16047 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 5 FOLLOWER]: Leader election won for term 5
05:23:05.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.878618 16047 raft_consensus.cc:686] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 5 LEADER]: Becoming Leader. State: Replica: a2f24784906a4e2ebb7a4d8073e17074, State: Running, Role: LEADER
05:23:05.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.879709 16047 consensus_queue.cc:237] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:05.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:05.888314 14894 catalog_manager.cc:5495] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 reported cstate change: term changed from 4 to 5, leader changed from 969c70d1882541a4b320cca5f9979912 (127.14.90.1) to a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2). New cstate: current_term: 5 leader_uuid: "a2f24784906a4e2ebb7a4d8073e17074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } health_report { overall_health: UNKNOWN } } }
05:23:06.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:06.300437 15985 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 5 FOLLOWER]: Refusing update from remote peer a2f24784906a4e2ebb7a4d8073e17074: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
05:23:06.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:06.302111 16047 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
05:23:06.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:06.344619 15736 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 -> Peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Couldn't send request to peer 969c70d1882541a4b320cca5f9979912. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:23:07.135 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:07.136 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:07.137 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=2457), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [3ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [15ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [17ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [20ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [21ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [35ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [37ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [38ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [38ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [77ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [80ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [83ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [84ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [131ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [135ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [136ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [138ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [138ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [435ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [436ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [439ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [440ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [2452ms] refreshing cache from master, [2452ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [2456ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@705369251(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2fa37815 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@993419682 after retry RPC@1758787704) -> (continuation of Deferred@476314037 after retry RPC@1523318892) -> (continuation of Deferred@1471830855 after retry RPC@1833959100) -> (continuation of Deferred@1628293648 after retry RPC@1420071031) -> (continuation of Deferred@1342506409 after retry RPC@228335247), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a893713 -> wakeup thread Time-limited test -> (continuation of Deferred@993419682 after retry RPC@1758787704) -> (continuation of Deferred@476314037 after retry RPC@1523318892) -> (continuation of Deferred@1471830855 after retry RPC@1833959100) -> (continuation of Deferred@1628293648 after retry RPC@1420071031) -> (continuation of Deferred@1342506409 after retry RPC@228335247))) after lookup
05:23:07.138 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:07.139 [DEBUG - kudu-nio-1] (Connection.java:757) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] Successfully connected to peer
05:23:07.172 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xa7556dfb, L:/127.0.0.1:47624 - R:/127.14.90.2:45685] using TOKEN/null
05:23:07.225 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.14.90.1:46843
05:23:07.226 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:23:07.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:07.227464 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:07.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/logs
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.1:46843
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.1
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34663
05:23:07.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.1
05:23:07.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:07.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:07.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:07.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:23:07.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:07.568215 16083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:23:07.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:07.568984 16083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:23:07.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:07.569615 16083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:23:07.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:07.604635 16083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:23:07.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:07.605664 16083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.1
05:23:07.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:07.651427 16083 tablet_server_runner.cc:78] Tablet server non-default flags:
05:23:07.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.1:46843
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.1
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34663
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:23:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.1
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/logs
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:23:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:07.652979 16083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:23:07.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:07.655046 16083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:23:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:07.675745 16089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:08.869904 15736 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 -> Peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Couldn't send request to peer 969c70d1882541a4b320cca5f9979912. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
05:23:08.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:07.675833 16092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:08.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:07.676482 16090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:08.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:07.677850 16083 server_base.cc:1034] running on GCE node
05:23:08.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.910748 16083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:23:08.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.913028 16083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:23:08.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.914407 16083 hybrid_clock.cc:648] HybridClock initialized: now 1783488188914357 us; error 62 us; skew 500 ppm
05:23:08.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.915310 16083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:23:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.921763 16083 webserver.cc:460] Webserver started at http://127.14.90.1:34663/ using document root <none> and password file <none>
05:23:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.922729 16083 fs_manager.cc:359] Metadata directory not provided
05:23:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.922941 16083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:23:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.931277 16083 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.001s
05:23:08.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.936045 16104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:23:08.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.937318 16083 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
05:23:08.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.937649 16083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:08.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "969c70d1882541a4b320cca5f9979912"
05:23:08.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:42 on dist-test-slave-zq0g"
05:23:08.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:08.940219 16083 fs_report.cc:383] FS layout report
05:23:08.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:23:08.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:08.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/data
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:23:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:23:09.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.019131 16083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:23:09.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.020606 16083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:23:09.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.021049 16083 kserver.cc:163] Server-wide thread pool size limit: 3276
05:23:09.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.023998 16083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:23:09.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.030308 16111 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:23:09.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.038780 16083 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:23:09.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.039103 16083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
05:23:09.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.039419 16083 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:23:09.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.045073 16083 ts_tablet_manager.cc:610] Registered 1 tablets
05:23:09.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.045349 16083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.004s	sys 0.000s
05:23:09.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.045719 16111 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap starting.
05:23:09.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.129262 16111 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Log is configured to *not* fsync() on all Append() calls
05:23:09.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.196103 16111 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:23:09.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.197278 16111 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap complete.
05:23:09.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.198861 16111 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Time spent bootstrapping tablet: real 0.154s	user 0.101s	sys 0.050s
05:23:09.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.221681 16111 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:09.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.226014 16111 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969c70d1882541a4b320cca5f9979912, State: Initialized, Role: FOLLOWER
05:23:09.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.227569 16111 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:09.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.233788 16111 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Time spent starting tablet: real 0.035s	user 0.025s	sys 0.008s
05:23:09.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.248575 16083 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.1:46843
05:23:09.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.248857 16219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.1:46843 every 8 connection(s)
05:23:09.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.251646 16083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:23:09.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.262126 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 16083
05:23:09.263 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:09.295 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:23:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.294987 16220 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:23:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.295467 16220 heartbeater.cc:464] Registering TS with master...
05:23:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.296666 16220 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:23:09.300 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.14.90.2:45685
05:23:09.301 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:23:09.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.302161 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 15715
05:23:09.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.302034 14892 ts_manager.cc:194] Re-registered known tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:23:09.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.305764 14892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.1:56083
05:23:09.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.318500 16221 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:23:09.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.318887 16221 heartbeater.cc:464] Registering TS with master...
05:23:09.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.319621 16221 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:23:09.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.323328 14815 ts_manager.cc:194] Re-registered known tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:23:09.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.326061 16222 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:23:09.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.326453 16222 heartbeater.cc:464] Registering TS with master...
05:23:09.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.327224 16222 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:23:09.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:09.330829 14741 ts_manager.cc:194] Re-registered known tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:23:09.342 [DEBUG - kudu-nio-1] (Connection.java:239) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] handling channelInactive
05:23:09.342 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state READY due to: connection closed
05:23:09.342 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:09.343 [DEBUG - Time-limited test] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:09.345 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.345 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.345 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.355 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:09.355 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:09.356 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [11ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@213721458(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7db8880b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cfb356e -> wakeup thread Time-limited test)) after lookup
05:23:09.357 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:09.358 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.358 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.358 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.374 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:09.375 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:09.375 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [11ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [13ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [15ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [31ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@213721458(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7db8880b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cfb356e -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410))) after lookup
05:23:09.376 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:09.378 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.378 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.379 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.394 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:09.395 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:09.395 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [11ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [13ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [15ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [31ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [33ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [36ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [36ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [51ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@213721458(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7db8880b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410) -> (continuation of Deferred@30751531 after retry RPC@2035022946), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cfb356e -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410) -> (continuation of Deferred@30751531 after retry RPC@2035022946))) after lookup
05:23:09.396 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:09.397 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.397 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.397 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.455 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:09.455 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:09.456 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=113), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [11ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [13ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [15ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [31ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [33ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [36ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [36ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [51ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [52ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [54ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [54ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [107ms] refreshing cache from master, [108ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [111ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@213721458(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7db8880b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410) -> (continuation of Deferred@30751531 after retry RPC@2035022946) -> (continuation of Deferred@22274339 after retry RPC@1522467351), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cfb356e -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410) -> (continuation of Deferred@30751531 after retry RPC@2035022946) -> (continuation of Deferred@22274339 after retry RPC@1522467351))) after lookup
05:23:09.457 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:09.459 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.460 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.460 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.534 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:09.534 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:09.535 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=191), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [11ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [13ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [15ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [31ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [33ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [36ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [36ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [51ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [52ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [54ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [54ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [107ms] refreshing cache from master, [108ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [111ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [113ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [117ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [117ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [190ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@213721458(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7db8880b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410) -> (continuation of Deferred@30751531 after retry RPC@2035022946) -> (continuation of Deferred@22274339 after retry RPC@1522467351) -> (continuation of Deferred@284877940 after retry RPC@846717894), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cfb356e -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410) -> (continuation of Deferred@30751531 after retry RPC@2035022946) -> (continuation of Deferred@22274339 after retry RPC@1522467351) -> (continuation of Deferred@284877940 after retry RPC@846717894))) after lookup
05:23:09.536 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:09.537 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.537 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:09.538 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:10.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.310917 16220 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:23:10.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.594185 16231 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2f24784906a4e2ebb7a4d8073e17074)
05:23:10.595 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:10.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.594707 16231 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:10.595 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:10.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.596637 16231 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:10.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:10.614319 15917 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111)
05:23:10.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.627203 16174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "969c70d1882541a4b320cca5f9979912" is_pre_election: true
05:23:10.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.627912 16174 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db66d29aaa8408c8acbad8bde8e4ae3 in term 4.
05:23:10.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.629191 15915 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, 969c70d1882541a4b320cca5f9979912; no voters: a2f24784906a4e2ebb7a4d8073e17074
05:23:10.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.630081 16231 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 5 FOLLOWER]: Leader pre-election won for term 6
05:23:10.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.630391 16231 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2f24784906a4e2ebb7a4d8073e17074)
05:23:10.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.630745 16231 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 5 FOLLOWER]: Advancing to term 6
05:23:10.631 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1252), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [11ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [13ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [15ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [31ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [33ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [36ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [36ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [51ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [52ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [54ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [54ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [107ms] refreshing cache from master, [108ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [111ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [113ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [117ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [117ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [190ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [192ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [194ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [195ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [1247ms] refreshing cache from master, [1247ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1251ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@213721458(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7db8880b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410) -> (continuation of Deferred@30751531 after retry RPC@2035022946) -> (continuation of Deferred@22274339 after retry RPC@1522467351) -> (continuation of Deferred@284877940 after retry RPC@846717894) -> (continuation of Deferred@343634452 after retry RPC@1949767008), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cfb356e -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410) -> (continuation of Deferred@30751531 after retry RPC@2035022946) -> (continuation of Deferred@22274339 after retry RPC@1522467351) -> (continuation of Deferred@284877940 after retry RPC@846717894) -> (continuation of Deferred@343634452 after retry RPC@1949767008))) after lookup
05:23:10.634 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:10.636 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:10.636 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:10.636 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:10.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.637179 16231 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:10.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.638747 16231 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 6 election: Requested vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:10.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.639600 16174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "969c70d1882541a4b320cca5f9979912"
05:23:10.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.640166 16174 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 4 FOLLOWER]: Advancing to term 6
05:23:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:10.642711 15917 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111)
05:23:10.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.649271 16174 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db66d29aaa8408c8acbad8bde8e4ae3 in term 6.
05:23:10.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.650298 15915 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, 969c70d1882541a4b320cca5f9979912; no voters: a2f24784906a4e2ebb7a4d8073e17074
05:23:10.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.651160 16231 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 6 FOLLOWER]: Leader election won for term 6
05:23:10.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.652331 16231 raft_consensus.cc:686] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 6 LEADER]: Becoming Leader. State: Replica: 3db66d29aaa8408c8acbad8bde8e4ae3, State: Running, Role: LEADER
05:23:10.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.653183 16231 consensus_queue.cc:237] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:10.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:10.661983 14892 catalog_manager.cc:5495] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 reported cstate change: term changed from 5 to 6, leader changed from a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2) to 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3). New cstate: current_term: 6 leader_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } health_report { overall_health: HEALTHY } } }
05:23:11.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:11.059973 15917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111) [suppressed 2 similar messages]
05:23:11.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:11.062217 15917 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 -> Peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Couldn't send request to peer a2f24784906a4e2ebb7a4d8073e17074. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:23:11.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:11.227175 16174 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 6 FOLLOWER]: Refusing update from remote peer 3db66d29aaa8408c8acbad8bde8e4ae3: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
05:23:11.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:11.228791 16231 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
05:23:13.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:13.721071 15917 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 -> Peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Couldn't send request to peer a2f24784906a4e2ebb7a4d8073e17074. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
05:23:14.694 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:14.694 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:14.695 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=14, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=5352), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [11ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [13ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [15ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [15ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [31ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [33ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [36ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [36ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [51ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [52ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [54ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [54ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [107ms] refreshing cache from master, [108ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [111ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [113ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [117ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [117ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [190ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [192ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [194ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [195ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [1247ms] refreshing cache from master, [1247ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1251ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [1289ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [1293ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [1293ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [5347ms] refreshing cache from master, [5347ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [5351ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@213721458(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7db8880b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410) -> (continuation of Deferred@30751531 after retry RPC@2035022946) -> (continuation of Deferred@22274339 after retry RPC@1522467351) -> (continuation of Deferred@284877940 after retry RPC@846717894) -> (continuation of Deferred@343634452 after retry RPC@1949767008) -> (continuation of Deferred@318963483 after retry RPC@1607627790), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cfb356e -> wakeup thread Time-limited test -> (continuation of Deferred@213590249 after retry RPC@434905410) -> (continuation of Deferred@30751531 after retry RPC@2035022946) -> (continuation of Deferred@22274339 after retry RPC@1522467351) -> (continuation of Deferred@284877940 after retry RPC@846717894) -> (continuation of Deferred@343634452 after retry RPC@1949767008) -> (continuation of Deferred@318963483 after retry RPC@1607627790))) after lookup
05:23:14.696 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:14.697 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] Successfully connected to peer
05:23:14.724 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x5f128b4c, L:/127.0.0.1:52782 - R:/127.14.90.3:35001] using TOKEN/null
05:23:14.777 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.14.90.2:45685
05:23:14.779 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:14.781167 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/logs
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.2:45685
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.2
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34839
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.2
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:23:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:15.107307 16274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:23:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:15.107795 16274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:23:15.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:15.108249 16274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:23:15.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:15.142765 16274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:23:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:15.143989 16274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.2
05:23:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:15.181298 16274 tablet_server_runner.cc:78] Tablet server non-default flags:
05:23:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:23:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data
05:23:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.2:45685
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.2
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34839
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.2
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/logs
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:23:15.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:23:15.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:23:15.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:23:15.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:23:15.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:23:15.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:23:15.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:23:15.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:23:15.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:15.182622 16274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:23:15.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:15.184805 16274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:23:15.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:15.209755 16281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:16.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:16.130398 15917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111) [suppressed 9 similar messages]
05:23:16.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:16.133028 15917 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 -> Peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Couldn't send request to peer a2f24784906a4e2ebb7a4d8073e17074. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
05:23:16.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:15.210968 16280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:16.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:15.211321 16274 server_base.cc:1034] running on GCE node
05:23:16.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:15.212396 16283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:16.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.477185 16274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:23:16.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.479816 16274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:23:16.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.481338 16274 hybrid_clock.cc:648] HybridClock initialized: now 1783488196481302 us; error 42 us; skew 500 ppm
05:23:16.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.482292 16274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:23:16.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.493366 16274 webserver.cc:460] Webserver started at http://127.14.90.2:34839/ using document root <none> and password file <none>
05:23:16.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.494397 16274 fs_manager.cc:359] Metadata directory not provided
05:23:16.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.494719 16274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:23:16.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.503317 16274 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.004s
05:23:16.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.508147 16294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:23:16.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.509413 16274 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
05:23:16.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.509702 16274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:16.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a2f24784906a4e2ebb7a4d8073e17074"
05:23:16.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:45 on dist-test-slave-zq0g"
05:23:16.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.512355 16274 fs_report.cc:383] FS layout report
05:23:16.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:23:16.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:16.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:16.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/data
05:23:16.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:23:16.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:23:16.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:23:16.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:23:16.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:23:16.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:23:16.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:23:16.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:23:16.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:23:16.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:23:16.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:23:16.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.559518 16274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:23:16.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.561378 16274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:23:16.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.561918 16274 kserver.cc:163] Server-wide thread pool size limit: 3276
05:23:16.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.564816 16274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:23:16.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.572408 16301 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:23:16.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.580595 16274 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:23:16.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.580905 16274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
05:23:16.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.581270 16274 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:23:16.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.586203 16274 ts_tablet_manager.cc:610] Registered 1 tablets
05:23:16.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.586447 16274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.002s	sys 0.000s
05:23:16.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.586946 16301 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Bootstrap starting.
05:23:16.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.650417 16301 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Log is configured to *not* fsync() on all Append() calls
05:23:16.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.756910 16301 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:23:16.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.758261 16301 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Bootstrap complete.
05:23:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.761003 16301 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Time spent bootstrapping tablet: real 0.175s	user 0.142s	sys 0.028s
05:23:16.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.771674 16274 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.2:45685
05:23:16.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.772357 16408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.2:45685 every 8 connection(s)
05:23:16.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.775667 16274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:23:16.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:16.784927 14772 debug-util.cc:398] Leaking SignalData structure 0x7b0800083fe0 after lost signal to thread 14709
05:23:16.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:16.785797 14772 debug-util.cc:398] Leaking SignalData structure 0x7b08000a2be0 after lost signal to thread 14775
05:23:16.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.785836 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 16274
05:23:16.789 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:16.796 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:97)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
05:23:16.797 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:79)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
05:23:16.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.795625 16301 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:16.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.799752 16301 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f24784906a4e2ebb7a4d8073e17074, State: Initialized, Role: FOLLOWER
05:23:16.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.800956 16301 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:16.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.827873 16301 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Time spent starting tablet: real 0.066s	user 0.032s	sys 0.016s
05:23:16.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.852825 16410 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:23:16.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.853518 16409 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:23:16.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.854012 16409 heartbeater.cc:464] Registering TS with master...
05:23:16.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.855726 16409 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:23:16.862 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:23:16.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.860616 14892 ts_manager.cc:194] Re-registered known tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:16.863 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.14.90.3:35001
05:23:16.863 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:23:16.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.862967 16410 heartbeater.cc:464] Registering TS with master...
05:23:16.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.863909 16410 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:23:16.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.864614 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 15890
05:23:16.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.867008 14892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.2:34157
05:23:16.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.869037 14815 ts_manager.cc:194] Re-registered known tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:16.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.874682 16409 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:23:16.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.880951 16411 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:23:16.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.881362 16411 heartbeater.cc:464] Registering TS with master...
05:23:16.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.882138 16411 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:23:16.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:16.886557 14740 ts_manager.cc:194] Re-registered known tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:16.901 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] handling channelInactive
05:23:16.901 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state READY due to: connection closed
05:23:16.901 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:16.902 [DEBUG - Time-limited test] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:16.903 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:16.904 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:16.904 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:16.915 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:16.916 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:16.916 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [9ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [14ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1352894028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3f9bb993 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7a4c9178 -> wakeup thread Time-limited test)) after lookup
05:23:16.918 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:16.919 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:16.919 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:16.919 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:16.935 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:16.935 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:16.935 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [9ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [14ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [16ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [18ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [33ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1352894028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3f9bb993 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1256515059 after retry RPC@803313378), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7a4c9178 -> wakeup thread Time-limited test -> (continuation of Deferred@1256515059 after retry RPC@803313378))) after lookup
05:23:16.936 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:16.938 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:16.938 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:16.938 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:16.975 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:16.975 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:16.975 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=74), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [9ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [14ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [16ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [18ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [33ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [35ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [37ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [73ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1352894028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3f9bb993 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1256515059 after retry RPC@803313378) -> (continuation of Deferred@275470383 after retry RPC@866028102), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7a4c9178 -> wakeup thread Time-limited test -> (continuation of Deferred@1256515059 after retry RPC@803313378) -> (continuation of Deferred@275470383 after retry RPC@866028102))) after lookup
05:23:16.977 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:16.978 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:16.978 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:16.979 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:17.035 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:17.036 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:17.036 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=135), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [9ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [14ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [16ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [18ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [33ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [35ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [37ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [73ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [75ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [77ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [78ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [130ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [134ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1352894028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3f9bb993 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1256515059 after retry RPC@803313378) -> (continuation of Deferred@275470383 after retry RPC@866028102) -> (continuation of Deferred@568709706 after retry RPC@1312345116), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7a4c9178 -> wakeup thread Time-limited test -> (continuation of Deferred@1256515059 after retry RPC@803313378) -> (continuation of Deferred@275470383 after retry RPC@866028102) -> (continuation of Deferred@568709706 after retry RPC@1312345116))) after lookup
05:23:17.037 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:17.039 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:17.039 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:17.040 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:17.514 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:17.514 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:17.515 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=613), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [9ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [14ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [16ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [18ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [33ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [35ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [37ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [73ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [75ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [77ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [78ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [130ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [134ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [136ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [139ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [139ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [609ms] refreshing cache from master, [609ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [613ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1352894028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3f9bb993 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1256515059 after retry RPC@803313378) -> (continuation of Deferred@275470383 after retry RPC@866028102) -> (continuation of Deferred@568709706 after retry RPC@1312345116) -> (continuation of Deferred@1572878678 after retry RPC@1427134174), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7a4c9178 -> wakeup thread Time-limited test -> (continuation of Deferred@1256515059 after retry RPC@803313378) -> (continuation of Deferred@275470383 after retry RPC@866028102) -> (continuation of Deferred@568709706 after retry RPC@1312345116) -> (continuation of Deferred@1572878678 after retry RPC@1427134174))) after lookup
05:23:17.516 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:17.517 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:17.517 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:17.517 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:18.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.184299 16423 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3db66d29aaa8408c8acbad8bde8e4ae3)
05:23:18.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.185030 16423 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:18.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.188126 16423 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:18.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:18.203604 16105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111)
05:23:18.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:18.212945 16105 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001): Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111)
05:23:18.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.217190 16364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "969c70d1882541a4b320cca5f9979912" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "a2f24784906a4e2ebb7a4d8073e17074" is_pre_election: true
05:23:18.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.218168 16364 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 969c70d1882541a4b320cca5f9979912 in term 5.
05:23:18.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.219854 16107 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 969c70d1882541a4b320cca5f9979912, a2f24784906a4e2ebb7a4d8073e17074; no voters: 3db66d29aaa8408c8acbad8bde8e4ae3
05:23:18.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.220899 16423 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 6 FOLLOWER]: Leader pre-election won for term 7
05:23:18.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.221268 16423 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3db66d29aaa8408c8acbad8bde8e4ae3)
05:23:18.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.221594 16423 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 6 FOLLOWER]: Advancing to term 7
05:23:18.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.228688 16423 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:18.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.231659 16364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "969c70d1882541a4b320cca5f9979912" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "a2f24784906a4e2ebb7a4d8073e17074"
05:23:18.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.232311 16364 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 5 FOLLOWER]: Advancing to term 7
05:23:18.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.232869 16423 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 7 election: Requested vote from peers a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:18.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:18.236940 16105 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001): Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111)
05:23:18.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.243281 16364 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 969c70d1882541a4b320cca5f9979912 in term 7.
05:23:18.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.244470 16107 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 969c70d1882541a4b320cca5f9979912, a2f24784906a4e2ebb7a4d8073e17074; no voters: 3db66d29aaa8408c8acbad8bde8e4ae3
05:23:18.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.245326 16423 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 7 FOLLOWER]: Leader election won for term 7
05:23:18.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.263065 16423 raft_consensus.cc:686] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 7 LEADER]: Becoming Leader. State: Replica: 969c70d1882541a4b320cca5f9979912, State: Running, Role: LEADER
05:23:18.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.264605 16423 consensus_queue.cc:237] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:18.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.277235 14892 catalog_manager.cc:5495] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 reported cstate change: term changed from 6 to 7, leader changed from 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3) to 969c70d1882541a4b320cca5f9979912 (127.14.90.1). New cstate: current_term: 7 leader_uuid: "969c70d1882541a4b320cca5f9979912" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } health_report { overall_health: UNKNOWN } } }
05:23:18.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.688748 16364 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 7 FOLLOWER]: Refusing update from remote peer 969c70d1882541a4b320cca5f9979912: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
05:23:18.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:18.690328 16423 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
05:23:18.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:18.725342 16105 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 -> Peer 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001): Couldn't send request to peer 3db66d29aaa8408c8acbad8bde8e4ae3. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:23:19.575 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:19.576 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:19.576 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2675), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [9ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [14ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [16ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [18ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [18ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [33ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [35ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [37ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [73ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [75ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [77ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [78ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [130ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [134ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [136ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [139ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [139ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [609ms] refreshing cache from master, [609ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [613ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [614ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [616ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [616ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [2669ms] refreshing cache from master, [2669ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [2674ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1352894028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3f9bb993 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1256515059 after retry RPC@803313378) -> (continuation of Deferred@275470383 after retry RPC@866028102) -> (continuation of Deferred@568709706 after retry RPC@1312345116) -> (continuation of Deferred@1572878678 after retry RPC@1427134174) -> (continuation of Deferred@325366169 after retry RPC@1828901219), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7a4c9178 -> wakeup thread Time-limited test -> (continuation of Deferred@1256515059 after retry RPC@803313378) -> (continuation of Deferred@275470383 after retry RPC@866028102) -> (continuation of Deferred@568709706 after retry RPC@1312345116) -> (continuation of Deferred@1572878678 after retry RPC@1427134174) -> (continuation of Deferred@325366169 after retry RPC@1828901219))) after lookup
05:23:19.577 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:19.580 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] Successfully connected to peer
05:23:19.601 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x76026e1c, L:/127.0.0.1:55802 - R:/127.14.90.1:46843] using TOKEN/null
05:23:19.658 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.14.90.3:35001
05:23:19.659 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:19.660892 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/logs
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:23:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.3:35001
05:23:19.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.3
05:23:19.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:19.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37591
05:23:19.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.3
05:23:19.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:19.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:19.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:19.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:23:19.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:19.988646 16455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:23:19.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:19.989133 16455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:23:19.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:19.989560 16455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:23:20.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:20.021456 16455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:23:20.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:20.022275 16455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.3
05:23:20.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:20.059367 16455 tablet_server_runner.cc:78] Tablet server non-default flags:
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.3:35001
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.3
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37591
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.3
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/logs
05:23:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:20.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:23:20.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:23:20.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:23:20.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:23:20.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:23:20.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:23:20.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:23:20.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:23:20.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:23:20.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:20.060825 16455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:23:20.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:20.062986 16455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:23:20.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:20.083978 16462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:21.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:21.201184 16105 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 -> Peer 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001): Couldn't send request to peer 3db66d29aaa8408c8acbad8bde8e4ae3. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
05:23:21.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:20.084079 16464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:21.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:20.085544 16461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:21.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:21.313189 16463 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
05:23:21.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.313319 16455 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
05:23:21.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.317426 16455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:23:21.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.319695 16455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:23:21.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.321066 16455 hybrid_clock.cc:648] HybridClock initialized: now 1783488201321043 us; error 46 us; skew 500 ppm
05:23:21.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.321838 16455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:23:21.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.328393 16455 webserver.cc:460] Webserver started at http://127.14.90.3:37591/ using document root <none> and password file <none>
05:23:21.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.329375 16455 fs_manager.cc:359] Metadata directory not provided
05:23:21.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.329578 16455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:23:21.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.338364 16455 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.006s	sys 0.000s
05:23:21.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.343421 16475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:23:21.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.344744 16455 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
05:23:21.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.345121 16455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:21.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:23:21.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:46 on dist-test-slave-zq0g"
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.347558 16455 fs_report.cc:383] FS layout report
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/data
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:23:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:23:21.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.388007 16455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:23:21.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.389554 16455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:23:21.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.390069 16455 kserver.cc:163] Server-wide thread pool size limit: 3276
05:23:21.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.392817 16455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:23:21.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.399164 16482 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:23:21.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.410254 16455 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:23:21.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.410586 16455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s	user 0.000s	sys 0.002s
05:23:21.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.410962 16455 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:23:21.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.416150 16455 ts_tablet_manager.cc:610] Registered 1 tablets
05:23:21.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.416409 16455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
05:23:21.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.416919 16482 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Bootstrap starting.
05:23:21.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.490415 16482 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Log is configured to *not* fsync() on all Append() calls
05:23:21.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.591941 16482 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:23:21.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.592720 16482 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Bootstrap complete.
05:23:21.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.594698 16482 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Time spent bootstrapping tablet: real 0.178s	user 0.119s	sys 0.055s
05:23:21.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.612892 16455 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.3:35001
05:23:21.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.613144 16589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.3:35001 every 8 connection(s)
05:23:21.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.616050 16455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:23:21.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.615113 16482 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:21.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.617785 16482 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db66d29aaa8408c8acbad8bde8e4ae3, State: Initialized, Role: FOLLOWER
05:23:21.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.618916 16482 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:21.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.619884 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 16455
05:23:21.621 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:21.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.667946 16482 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Time spent starting tablet: real 0.073s	user 0.019s	sys 0.035s
05:23:21.688 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:23:21.689 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.14.90.1:46843
05:23:21.689 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:23:21.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.687153 16590 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:23:21.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.687640 16590 heartbeater.cc:464] Registering TS with master...
05:23:21.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.688771 16590 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:23:21.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.692121 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 16083
05:23:21.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.692494 16591 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:23:21.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.692848 16591 heartbeater.cc:464] Registering TS with master...
05:23:21.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.692746 14892 ts_manager.cc:194] Re-registered known tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:21.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.693727 16591 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:23:21.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.694921 16596 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:23:21.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.695257 16596 heartbeater.cc:464] Registering TS with master...
05:23:21.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.696084 16596 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:23:21.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.696175 14892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.3:57535
05:23:21.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.700855 14815 ts_manager.cc:194] Re-registered known tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:21.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.700822 16590 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:23:21.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:21.702421 14740 ts_manager.cc:194] Re-registered known tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:21.725 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] handling channelInactive
05:23:21.725 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state READY due to: connection closed
05:23:21.726 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:21.726 [DEBUG - Time-limited test] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:21.728 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.728 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.729 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.754 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:21.754 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:21.755 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@868809242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@d656f30 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@71e9c9b2 -> wakeup thread Time-limited test)) after lookup
05:23:21.756 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:21.758 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.758 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.759 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.775 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:21.775 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:21.775 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=49), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [30ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [33ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [33ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [48ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@868809242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@d656f30 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@71e9c9b2 -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558))) after lookup
05:23:21.776 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:21.777 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.777 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.777 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.815 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:21.815 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:21.816 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=89), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [30ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [33ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [33ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [48ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [50ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [51ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [51ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [88ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@868809242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@d656f30 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558) -> (continuation of Deferred@568307443 after retry RPC@1242820123), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@71e9c9b2 -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558) -> (continuation of Deferred@568307443 after retry RPC@1242820123))) after lookup
05:23:21.817 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:21.818 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.818 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.819 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.834 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:21.834 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:21.835 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=108), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [30ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [33ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [33ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [48ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [50ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [51ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [51ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [88ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [90ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [93ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [93ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [104ms] refreshing cache from master, [104ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [108ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@868809242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@d656f30 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558) -> (continuation of Deferred@568307443 after retry RPC@1242820123) -> (continuation of Deferred@250822010 after retry RPC@1539664265), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@71e9c9b2 -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558) -> (continuation of Deferred@568307443 after retry RPC@1242820123) -> (continuation of Deferred@250822010 after retry RPC@1539664265))) after lookup
05:23:21.835 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:21.836 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.836 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:21.836 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:22.014 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:22.014 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:22.015 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=288), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [30ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [33ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [33ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [48ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [50ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [51ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [51ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [88ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [90ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [93ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [93ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [104ms] refreshing cache from master, [104ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [108ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [109ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [110ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [110ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [284ms] refreshing cache from master, [284ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [287ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@868809242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@d656f30 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558) -> (continuation of Deferred@568307443 after retry RPC@1242820123) -> (continuation of Deferred@250822010 after retry RPC@1539664265) -> (continuation of Deferred@1275202388 after retry RPC@1016538318), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@71e9c9b2 -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558) -> (continuation of Deferred@568307443 after retry RPC@1242820123) -> (continuation of Deferred@250822010 after retry RPC@1539664265) -> (continuation of Deferred@1275202388 after retry RPC@1016538318))) after lookup
05:23:22.016 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:22.018 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:22.018 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:22.018 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:22.755 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:22.755 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:22.756 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1029), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [30ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [33ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [33ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [48ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [50ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [51ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [51ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [88ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [90ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [93ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [93ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [104ms] refreshing cache from master, [104ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [108ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [109ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [110ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [110ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [284ms] refreshing cache from master, [284ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [287ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [289ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [292ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [292ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1024ms] refreshing cache from master, [1024ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1028ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@868809242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@d656f30 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558) -> (continuation of Deferred@568307443 after retry RPC@1242820123) -> (continuation of Deferred@250822010 after retry RPC@1539664265) -> (continuation of Deferred@1275202388 after retry RPC@1016538318) -> (continuation of Deferred@1560918159 after retry RPC@1488803508), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@71e9c9b2 -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558) -> (continuation of Deferred@568307443 after retry RPC@1242820123) -> (continuation of Deferred@250822010 after retry RPC@1539664265) -> (continuation of Deferred@1275202388 after retry RPC@1016538318) -> (continuation of Deferred@1560918159 after retry RPC@1488803508))) after lookup
05:23:22.757 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:22.757 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:22.758 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:22.758 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:22.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.778918 16602 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 969c70d1882541a4b320cca5f9979912)
05:23:22.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.779381 16602 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:22.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.781523 16602 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:22.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:22.783728 16295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:23:22.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:22.788403 16295 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:23:22.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.795776 16545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" is_pre_election: true
05:23:22.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.796571 16545 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f24784906a4e2ebb7a4d8073e17074 in term 6.
05:23:22.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.798223 16295 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, a2f24784906a4e2ebb7a4d8073e17074; no voters: 969c70d1882541a4b320cca5f9979912
05:23:22.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.798993 16602 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 7 FOLLOWER]: Leader pre-election won for term 8
05:23:22.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.799360 16602 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 969c70d1882541a4b320cca5f9979912)
05:23:22.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.799934 16602 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 7 FOLLOWER]: Advancing to term 8
05:23:22.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.804380 16602 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:22.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.805850 16602 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 8 election: Requested vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:22.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.807601 16545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:23:22.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.808168 16545 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 6 FOLLOWER]: Advancing to term 8
05:23:22.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:22.809333 16295 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:23:22.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.814947 16545 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f24784906a4e2ebb7a4d8073e17074 in term 8.
05:23:22.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.815860 16295 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, a2f24784906a4e2ebb7a4d8073e17074; no voters: 969c70d1882541a4b320cca5f9979912
05:23:22.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.816565 16602 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 8 FOLLOWER]: Leader election won for term 8
05:23:22.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.818599 16602 raft_consensus.cc:686] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 8 LEADER]: Becoming Leader. State: Replica: a2f24784906a4e2ebb7a4d8073e17074, State: Running, Role: LEADER
05:23:22.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.819583 16602 consensus_queue.cc:237] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:22.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:22.829070 14894 catalog_manager.cc:5495] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 reported cstate change: term changed from 7 to 8, leader changed from 969c70d1882541a4b320cca5f9979912 (127.14.90.1) to a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2). New cstate: current_term: 8 leader_uuid: "a2f24784906a4e2ebb7a4d8073e17074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } health_report { overall_health: UNKNOWN } } }
05:23:23.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:23.208413 16545 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 8 FOLLOWER]: Refusing update from remote peer a2f24784906a4e2ebb7a4d8073e17074: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
05:23:23.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:23.209802 16602 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
05:23:23.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:23.248492 16295 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 -> Peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Couldn't send request to peer 969c70d1882541a4b320cca5f9979912. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:23:25.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:25.790462 16295 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 -> Peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Couldn't send request to peer 969c70d1882541a4b320cca5f9979912. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
05:23:26.054 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:26.055 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:26.056 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=4329), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [3ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [28ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [30ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [33ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [33ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [48ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [50ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [51ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [51ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [88ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [90ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [93ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [93ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [104ms] refreshing cache from master, [104ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [108ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [109ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [110ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [110ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [284ms] refreshing cache from master, [284ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [287ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [289ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [292ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [292ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1024ms] refreshing cache from master, [1024ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1028ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [1030ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [1032ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1032ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [4324ms] refreshing cache from master, [4324ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [4328ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@868809242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@d656f30 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558) -> (continuation of Deferred@568307443 after retry RPC@1242820123) -> (continuation of Deferred@250822010 after retry RPC@1539664265) -> (continuation of Deferred@1275202388 after retry RPC@1016538318) -> (continuation of Deferred@1560918159 after retry RPC@1488803508) -> (continuation of Deferred@1947123168 after retry RPC@270337325), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@71e9c9b2 -> wakeup thread Time-limited test -> (continuation of Deferred@917095536 after retry RPC@162676558) -> (continuation of Deferred@568307443 after retry RPC@1242820123) -> (continuation of Deferred@250822010 after retry RPC@1539664265) -> (continuation of Deferred@1275202388 after retry RPC@1016538318) -> (continuation of Deferred@1560918159 after retry RPC@1488803508) -> (continuation of Deferred@1947123168 after retry RPC@270337325))) after lookup
05:23:26.057 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:26.059 [DEBUG - kudu-nio-0] (Connection.java:757) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] Successfully connected to peer
05:23:26.080 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x69fd7433, L:/127.0.0.1:51730 - R:/127.14.90.2:45685] using TOKEN/null
05:23:26.135 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.14.90.1:46843
05:23:26.135 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:26.137035 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/logs
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:26.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.1:46843
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.1
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34663
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.1
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:26.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:23:26.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:26.465018 16647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:23:26.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:26.465526 16647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:23:26.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:26.465972 16647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:23:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:26.501825 16647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:23:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:26.502892 16647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.1
05:23:26.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:26.541051 16647 tablet_server_runner.cc:78] Tablet server non-default flags:
05:23:26.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:26.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:26.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.1:46843
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.1
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34663
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.1
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/logs
05:23:26.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:23:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:23:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:23:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:23:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:23:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:23:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:23:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:23:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:23:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:26.543128 16647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:23:26.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:26.545007 16647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:23:26.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:26.568799 16658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:26.568993 16655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:27.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:26.569020 16656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:27.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:26.570714 16647 server_base.cc:1034] running on GCE node
05:23:27.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.797631 16647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:23:27.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.801018 16647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:23:27.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.802723 16647 hybrid_clock.cc:648] HybridClock initialized: now 1783488207802682 us; error 69 us; skew 500 ppm
05:23:27.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.803599 16647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:23:27.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.820434 16647 webserver.cc:460] Webserver started at http://127.14.90.1:34663/ using document root <none> and password file <none>
05:23:27.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.822037 16647 fs_manager.cc:359] Metadata directory not provided
05:23:27.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.822391 16647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:23:27.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.831017 16647 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.000s
05:23:27.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.835987 16668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:23:27.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.837136 16647 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
05:23:27.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.837549 16647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:27.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "969c70d1882541a4b320cca5f9979912"
05:23:27.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:42 on dist-test-slave-zq0g"
05:23:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.840353 16647 fs_report.cc:383] FS layout report
05:23:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:23:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/data
05:23:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:23:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:23:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:23:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:23:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:23:27.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:23:27.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:23:27.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:23:27.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:23:27.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:23:27.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:23:27.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:27.880460 16295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111) [suppressed 10 similar messages]
05:23:27.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.885461 16647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:23:27.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.887013 16647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:23:27.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.887428 16647 kserver.cc:163] Server-wide thread pool size limit: 3276
05:23:27.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.889950 16647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:23:27.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.895949 16677 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:23:27.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.904162 16647 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:23:27.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.904501 16647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
05:23:27.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.904875 16647 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:23:27.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.910341 16647 ts_tablet_manager.cc:610] Registered 1 tablets
05:23:27.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.910650 16647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.005s	sys 0.001s
05:23:27.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.911090 16677 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap starting.
05:23:27.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:27.982177 16677 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Log is configured to *not* fsync() on all Append() calls
05:23:28.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.071309 16677 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:23:28.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.072121 16677 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap complete.
05:23:28.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.074079 16677 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Time spent bootstrapping tablet: real 0.163s	user 0.100s	sys 0.057s
05:23:28.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.094337 16677 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:28.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.097589 16677 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969c70d1882541a4b320cca5f9979912, State: Initialized, Role: FOLLOWER
05:23:28.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.098634 16677 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:28.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.103224 16677 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Time spent starting tablet: real 0.029s	user 0.024s	sys 0.008s
05:23:28.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.110497 16647 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.1:46843
05:23:28.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.110702 16785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.1:46843 every 8 connection(s)
05:23:28.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.113171 16647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:23:28.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.122133 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 16647
05:23:28.123 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:28.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.173687 16787 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:23:28.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.174681 16787 heartbeater.cc:464] Registering TS with master...
05:23:28.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.175885 16787 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:23:28.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.181140 14815 ts_manager.cc:194] Re-registered known tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:23:28.189 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:23:28.191 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.14.90.2:45685
05:23:28.192 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:23:28.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.192580 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 16274
05:23:28.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.197172 16786 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:23:28.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.197611 16786 heartbeater.cc:464] Registering TS with master...
05:23:28.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.198983 16786 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:23:28.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.203661 14892 ts_manager.cc:194] Re-registered known tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:23:28.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.208958 14892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.1:58533
05:23:28.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.209218 16788 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:23:28.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.209571 16788 heartbeater.cc:464] Registering TS with master...
05:23:28.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.210434 16788 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:23:28.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:28.214210 14740 ts_manager.cc:194] Re-registered known tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:23:28.231 [DEBUG - kudu-nio-0] (Connection.java:239) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] handling channelInactive
05:23:28.231 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state READY due to: connection closed
05:23:28.232 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:28.233 [DEBUG - Time-limited test] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:28.235 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.235 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.235 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.255 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:28.256 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:28.256 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1672014658(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7d6d64bb -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@605cad46 -> wakeup thread Time-limited test)) after lookup
05:23:28.257 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:28.259 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.259 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.259 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.276 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:28.277 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:28.277 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [23ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [26ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [26ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [43ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1672014658(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7d6d64bb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1426965337 after retry RPC@1354658339), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@605cad46 -> wakeup thread Time-limited test -> (continuation of Deferred@1426965337 after retry RPC@1354658339))) after lookup
05:23:28.278 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:28.279 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.279 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.279 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.295 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:28.295 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:28.296 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=62), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [23ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [26ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [26ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [43ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [45ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [46ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [47ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [61ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1672014658(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7d6d64bb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1426965337 after retry RPC@1354658339) -> (continuation of Deferred@1964604731 after retry RPC@132777990), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@605cad46 -> wakeup thread Time-limited test -> (continuation of Deferred@1426965337 after retry RPC@1354658339) -> (continuation of Deferred@1964604731 after retry RPC@132777990))) after lookup
05:23:28.296 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:28.298 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.299 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.299 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.413 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:28.414 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:28.414 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=181), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [23ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [26ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [26ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [43ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [45ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [46ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [47ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [61ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [63ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [66ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [66ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [180ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1672014658(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7d6d64bb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1426965337 after retry RPC@1354658339) -> (continuation of Deferred@1964604731 after retry RPC@132777990) -> (continuation of Deferred@2025648942 after retry RPC@792821255), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@605cad46 -> wakeup thread Time-limited test -> (continuation of Deferred@1426965337 after retry RPC@1354658339) -> (continuation of Deferred@1964604731 after retry RPC@132777990) -> (continuation of Deferred@2025648942 after retry RPC@792821255))) after lookup
05:23:28.415 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:28.416 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.416 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.416 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.573 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:28.574 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:28.574 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=341), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [23ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [26ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [26ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [43ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [45ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [46ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [47ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [61ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [63ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [66ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [66ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [180ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [182ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [183ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [183ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [337ms] refreshing cache from master, [337ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [340ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1672014658(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7d6d64bb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1426965337 after retry RPC@1354658339) -> (continuation of Deferred@1964604731 after retry RPC@132777990) -> (continuation of Deferred@2025648942 after retry RPC@792821255) -> (continuation of Deferred@1767774474 after retry RPC@1875734475), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@605cad46 -> wakeup thread Time-limited test -> (continuation of Deferred@1426965337 after retry RPC@1354658339) -> (continuation of Deferred@1964604731 after retry RPC@132777990) -> (continuation of Deferred@2025648942 after retry RPC@792821255) -> (continuation of Deferred@1767774474 after retry RPC@1875734475))) after lookup
05:23:28.575 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:28.577 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.577 [DEBUG - kudu-nio-0] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state CONNECTING due to: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:28.578 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685
05:23:29.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.215001 16786 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:23:29.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.243444 16797 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a2f24784906a4e2ebb7a4d8073e17074)
05:23:29.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.243973 16797 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:29.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.246237 16797 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:29.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:29.249902 16478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111)
05:23:29.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:29.255141 16478 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111)
05:23:29.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.261693 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "969c70d1882541a4b320cca5f9979912" is_pre_election: true
05:23:29.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.262548 16740 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db66d29aaa8408c8acbad8bde8e4ae3 in term 7.
05:23:29.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.263984 16476 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, 969c70d1882541a4b320cca5f9979912; no voters: a2f24784906a4e2ebb7a4d8073e17074
05:23:29.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.264766 16797 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 8 FOLLOWER]: Leader pre-election won for term 9
05:23:29.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.265110 16797 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a2f24784906a4e2ebb7a4d8073e17074)
05:23:29.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.265450 16797 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 8 FOLLOWER]: Advancing to term 9
05:23:29.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.270519 16797 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:29.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.272259 16797 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 9 election: Requested vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:29.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.272980 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "969c70d1882541a4b320cca5f9979912"
05:23:29.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.273494 16740 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 7 FOLLOWER]: Advancing to term 9
05:23:29.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:29.276719 16478 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111)
05:23:29.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.282235 16740 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db66d29aaa8408c8acbad8bde8e4ae3 in term 9.
05:23:29.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.283139 16476 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, 969c70d1882541a4b320cca5f9979912; no voters: a2f24784906a4e2ebb7a4d8073e17074
05:23:29.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.283748 16797 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 9 FOLLOWER]: Leader election won for term 9
05:23:29.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.285269 16797 raft_consensus.cc:686] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 9 LEADER]: Becoming Leader. State: Replica: 3db66d29aaa8408c8acbad8bde8e4ae3, State: Running, Role: LEADER
05:23:29.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.286193 16797 consensus_queue.cc:237] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:29.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.294819 14892 catalog_manager.cc:5495] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 reported cstate change: term changed from 8 to 9, leader changed from a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2) to 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3). New cstate: current_term: 9 leader_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } health_report { overall_health: HEALTHY } } }
05:23:29.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.687947 16740 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 9 FOLLOWER]: Refusing update from remote peer 3db66d29aaa8408c8acbad8bde8e4ae3: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
05:23:29.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:29.689478 16797 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
05:23:29.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:29.724304 16478 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 -> Peer a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685): Couldn't send request to peer a2f24784906a4e2ebb7a4d8073e17074. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.2:45685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:23:29.934 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:29.934 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:29.935 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1702), Traces: [0ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [2ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [2ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [22ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [23ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [26ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [26ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [43ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [45ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [46ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [47ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [61ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [63ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [66ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [66ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [180ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [182ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [183ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [183ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [337ms] refreshing cache from master, [337ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [340ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [342ms] sending RPC to server a2f24784906a4e2ebb7a4d8073e17074, [344ms] received response from server a2f24784906a4e2ebb7a4d8073e17074: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [345ms] delaying RPC due to: Network error: Failed to connect to peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685): Connection refused: /127.14.90.2:45685, [1697ms] refreshing cache from master, [1697ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1701ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@1672014658(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@7d6d64bb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1426965337 after retry RPC@1354658339) -> (continuation of Deferred@1964604731 after retry RPC@132777990) -> (continuation of Deferred@2025648942 after retry RPC@792821255) -> (continuation of Deferred@1767774474 after retry RPC@1875734475) -> (continuation of Deferred@568871289 after retry RPC@627038648), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@605cad46 -> wakeup thread Time-limited test -> (continuation of Deferred@1426965337 after retry RPC@1354658339) -> (continuation of Deferred@1964604731 after retry RPC@132777990) -> (continuation of Deferred@2025648942 after retry RPC@792821255) -> (continuation of Deferred@1767774474 after retry RPC@1875734475) -> (continuation of Deferred@568871289 after retry RPC@627038648))) after lookup
05:23:29.936 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:29.938 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] Successfully connected to peer
05:23:29.958 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x68a35ec0, L:/127.0.0.1:45926 - R:/127.14.90.3:35001] using TOKEN/null
05:23:30.025 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.14.90.2:45685
05:23:30.025 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:23:30.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:30.027158 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:30.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:30.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:30.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data
05:23:30.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:23:30.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:23:30.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:23:30.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:30.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/logs
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.2:45685
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.2
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34839
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.2
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:30.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:23:30.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:30.383355 16826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:23:30.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:30.383873 16826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:23:30.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:30.384320 16826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:23:30.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:30.417526 16826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:23:30.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:30.418390 16826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.2
05:23:30.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:30.456130 16826 tablet_server_runner.cc:78] Tablet server non-default flags:
05:23:30.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:30.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:30.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:30.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:23:30.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.2:45685
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.2
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34839
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:23:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.2
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/logs
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:23:30.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:30.457944 16826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:23:30.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:30.460201 16826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:23:30.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:30.481669 16835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:31.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:31.672349 14847 debug-util.cc:398] Leaking SignalData structure 0x7b08000677c0 after lost signal to thread 14782
05:23:31.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:31.673122 14847 debug-util.cc:398] Leaking SignalData structure 0x7b08000bb0c0 after lost signal to thread 14850
05:23:31.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:30.483337 16837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:31.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:30.485046 16826 server_base.cc:1034] running on GCE node
05:23:31.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:30.482247 16834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:31.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.764051 16826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:23:31.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.767176 16826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:23:31.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.770891 16826 hybrid_clock.cc:648] HybridClock initialized: now 1783488211770962 us; error 280 us; skew 500 ppm
05:23:31.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.771966 16826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:23:31.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.780839 16826 webserver.cc:460] Webserver started at http://127.14.90.2:34839/ using document root <none> and password file <none>
05:23:31.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.782351 16826 fs_manager.cc:359] Metadata directory not provided
05:23:31.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.782859 16826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:23:31.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.792984 16826 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.001s	sys 0.004s
05:23:31.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.799414 16848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:23:31.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.800964 16826 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.003s
05:23:31.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.801445 16826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:31.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a2f24784906a4e2ebb7a4d8073e17074"
05:23:31.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:45 on dist-test-slave-zq0g"
05:23:31.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.805706 16826 fs_report.cc:383] FS layout report
05:23:31.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:23:31.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:31.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/wal
05:23:31.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/data
05:23:31.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:23:31.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:23:31.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:23:31.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:23:31.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:23:31.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:23:31.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:23:31.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:23:31.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:23:31.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:23:31.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:23:31.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.859284 16826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:23:31.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.861526 16826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:23:31.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.862174 16826 kserver.cc:163] Server-wide thread pool size limit: 3276
05:23:31.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.866041 16826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:23:31.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.874993 16855 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:23:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.884441 16826 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:23:31.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.884843 16826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s	user 0.003s	sys 0.000s
05:23:31.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.885211 16826 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:23:31.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.893846 16826 ts_tablet_manager.cc:610] Registered 1 tablets
05:23:31.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.894164 16826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s	user 0.006s	sys 0.000s
05:23:31.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.894750 16855 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Bootstrap starting.
05:23:31.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:31.983000 16855 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Log is configured to *not* fsync() on all Append() calls
05:23:32.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.141733 16855 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:23:32.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.143813 16855 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Bootstrap complete.
05:23:32.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.146826 16855 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Time spent bootstrapping tablet: real 0.253s	user 0.190s	sys 0.050s
05:23:32.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.169678 16826 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.2:45685
05:23:32.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.170154 16964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.2:45685 every 8 connection(s)
05:23:32.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.173610 16826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-1/data/info.pb
05:23:32.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.183656 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 16826
05:23:32.184 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:32.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.180245 16855 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:32.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.185323 16855 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f24784906a4e2ebb7a4d8073e17074, State: Initialized, Role: FOLLOWER
05:23:32.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.189234 16855 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:32.226 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:23:32.227 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.14.90.3:35001
05:23:32.227 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:23:32.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.228250 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 16455
05:23:32.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.298408 16855 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074: Time spent starting tablet: real 0.151s	user 0.077s	sys 0.064s
05:23:32.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.304553 16965 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:23:32.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.305104 16965 heartbeater.cc:464] Registering TS with master...
05:23:32.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.306625 16965 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:23:32.307 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] handling channelInactive
05:23:32.308 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state READY due to: connection closed
05:23:32.310 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:32.311 [DEBUG - Time-limited test] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:32.312 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.313 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.313 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.310688 16966 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:23:32.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.311075 16966 heartbeater.cc:464] Registering TS with master...
05:23:32.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.311928 16966 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:23:32.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.315126 14894 ts_manager.cc:194] Re-registered known tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:32.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.318044 14894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.2:36383
05:23:32.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.320358 14815 ts_manager.cc:194] Re-registered known tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:32.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.323907 16971 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:23:32.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.324337 16971 heartbeater.cc:464] Registering TS with master...
05:23:32.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.325093 16971 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:23:32.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.324608 16965 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:23:32.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:32.329345 14740 ts_manager.cc:194] Re-registered known tserver with Master: a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685)
05:23:32.335 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:32.336 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:32.336 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [25ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@723096720(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2429ceea -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46f744a1 -> wakeup thread Time-limited test)) after lookup
05:23:32.337 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:32.338 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.338 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.338 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.354 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:32.355 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:32.355 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [25ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [26ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [28ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [28ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [44ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@723096720(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2429ceea -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2056096497 after retry RPC@1379159352), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46f744a1 -> wakeup thread Time-limited test -> (continuation of Deferred@2056096497 after retry RPC@1379159352))) after lookup
05:23:32.356 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:32.358 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.358 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.358 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.394 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:32.395 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:32.395 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=84), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [25ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [26ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [28ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [28ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [44ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [46ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [48ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [48ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [84ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@723096720(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2429ceea -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2056096497 after retry RPC@1379159352) -> (continuation of Deferred@1286933031 after retry RPC@1016648452), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46f744a1 -> wakeup thread Time-limited test -> (continuation of Deferred@2056096497 after retry RPC@1379159352) -> (continuation of Deferred@1286933031 after retry RPC@1016648452))) after lookup
05:23:32.396 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:32.397 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.397 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.397 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.515 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:32.515 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:32.515 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=205), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [25ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [26ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [28ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [28ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [44ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [46ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [48ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [48ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [84ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [86ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [87ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [87ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [200ms] refreshing cache from master, [200ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [204ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@723096720(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2429ceea -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2056096497 after retry RPC@1379159352) -> (continuation of Deferred@1286933031 after retry RPC@1016648452) -> (continuation of Deferred@1510832930 after retry RPC@405181880), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46f744a1 -> wakeup thread Time-limited test -> (continuation of Deferred@2056096497 after retry RPC@1379159352) -> (continuation of Deferred@1286933031 after retry RPC@1016648452) -> (continuation of Deferred@1510832930 after retry RPC@405181880))) after lookup
05:23:32.517 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:32.519 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.519 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.519 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.855 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:32.856 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)[L],969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:32.856 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=545), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [25ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [26ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [28ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [28ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [44ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [46ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [48ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [48ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [84ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [86ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [87ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [87ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [200ms] refreshing cache from master, [200ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [204ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [206ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [209ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [209ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [541ms] refreshing cache from master, [541ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [545ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@723096720(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2429ceea -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2056096497 after retry RPC@1379159352) -> (continuation of Deferred@1286933031 after retry RPC@1016648452) -> (continuation of Deferred@1510832930 after retry RPC@405181880) -> (continuation of Deferred@82881229 after retry RPC@685208766), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46f744a1 -> wakeup thread Time-limited test -> (continuation of Deferred@2056096497 after retry RPC@1379159352) -> (continuation of Deferred@1286933031 after retry RPC@1016648452) -> (continuation of Deferred@1510832930 after retry RPC@405181880) -> (continuation of Deferred@82881229 after retry RPC@685208766))) after lookup
05:23:32.857 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] connecting to peer
05:23:32.857 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.858 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:32.858 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001
05:23:33.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.655153 16977 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3db66d29aaa8408c8acbad8bde8e4ae3)
05:23:33.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.655637 16977 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:33.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.657917 16977 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:33.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:33.661029 16669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111)
05:23:33.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:33.664925 16669 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001): Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111)
05:23:33.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.673540 16920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "969c70d1882541a4b320cca5f9979912" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "a2f24784906a4e2ebb7a4d8073e17074" is_pre_election: true
05:23:33.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.674311 16920 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 969c70d1882541a4b320cca5f9979912 in term 8.
05:23:33.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.675638 16671 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 969c70d1882541a4b320cca5f9979912, a2f24784906a4e2ebb7a4d8073e17074; no voters: 3db66d29aaa8408c8acbad8bde8e4ae3
05:23:33.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.676564 16977 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 9 FOLLOWER]: Leader pre-election won for term 10
05:23:33.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.677076 16977 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3db66d29aaa8408c8acbad8bde8e4ae3)
05:23:33.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.677413 16977 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 9 FOLLOWER]: Advancing to term 10
05:23:33.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.682126 16977 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:33.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.683840 16977 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 10 election: Requested vote from peers a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2:45685), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:33.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.684825 16920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "969c70d1882541a4b320cca5f9979912" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "a2f24784906a4e2ebb7a4d8073e17074"
05:23:33.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.685384 16920 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 8 FOLLOWER]: Advancing to term 10
05:23:33.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:33.687487 16669 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001): Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111)
05:23:33.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.692912 16920 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 969c70d1882541a4b320cca5f9979912 in term 10.
05:23:33.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.693907 16671 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 969c70d1882541a4b320cca5f9979912, a2f24784906a4e2ebb7a4d8073e17074; no voters: 3db66d29aaa8408c8acbad8bde8e4ae3
05:23:33.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.694672 16977 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 10 FOLLOWER]: Leader election won for term 10
05:23:33.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.696311 16977 raft_consensus.cc:686] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 10 LEADER]: Becoming Leader. State: Replica: 969c70d1882541a4b320cca5f9979912, State: Running, Role: LEADER
05:23:33.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.697494 16977 consensus_queue.cc:237] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:33.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:33.706939 14894 catalog_manager.cc:5495] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 reported cstate change: term changed from 9 to 10, leader changed from 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3) to 969c70d1882541a4b320cca5f9979912 (127.14.90.1). New cstate: current_term: 10 leader_uuid: "969c70d1882541a4b320cca5f9979912" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } health_report { overall_health: UNKNOWN } } }
05:23:33.994 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:33.995 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:33.995 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1684), Traces: [0ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [3ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [25ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [26ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [28ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [28ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [44ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [46ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [48ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [48ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [84ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [86ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [87ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [87ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [200ms] refreshing cache from master, [200ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [204ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [206ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [209ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [209ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [541ms] refreshing cache from master, [541ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [545ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [546ms] sending RPC to server 3db66d29aaa8408c8acbad8bde8e4ae3, [548ms] received response from server 3db66d29aaa8408c8acbad8bde8e4ae3: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [548ms] delaying RPC due to: Network error: Failed to connect to peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001): Connection refused: /127.14.90.3:35001, [1680ms] refreshing cache from master, [1680ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1684ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@723096720(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@2429ceea -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2056096497 after retry RPC@1379159352) -> (continuation of Deferred@1286933031 after retry RPC@1016648452) -> (continuation of Deferred@1510832930 after retry RPC@405181880) -> (continuation of Deferred@82881229 after retry RPC@685208766) -> (continuation of Deferred@549682435 after retry RPC@207377017), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46f744a1 -> wakeup thread Time-limited test -> (continuation of Deferred@2056096497 after retry RPC@1379159352) -> (continuation of Deferred@1286933031 after retry RPC@1016648452) -> (continuation of Deferred@1510832930 after retry RPC@405181880) -> (continuation of Deferred@82881229 after retry RPC@685208766) -> (continuation of Deferred@549682435 after retry RPC@207377017))) after lookup
05:23:33.996 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:33.998 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] Successfully connected to peer
05:23:34.020 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xeecc2337, L:/127.0.0.1:57616 - R:/127.14.90.1:46843] using TOKEN/null
05:23:34.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:34.030856 16920 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 10 FOLLOWER]: Refusing update from remote peer 969c70d1882541a4b320cca5f9979912: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 38. (index mismatch)
05:23:34.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:34.032152 16669 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 -> Peer 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001): Couldn't send request to peer 3db66d29aaa8408c8acbad8bde8e4ae3. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.3:35001: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:23:34.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:34.032476 16982 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
05:23:34.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:34.061295 16998 mvcc.cc:205] Tried to move back new op lower bound from 7305167724651843584 to 7305167723314352128. Current Snapshot: MvccSnapshot[applied={T|T < 7305167724651843584}]
05:23:34.102 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.14.90.3:35001
05:23:34.103 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:23:34.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:34.104980 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:34.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:34.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:34.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data
05:23:34.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:23:34.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:23:34.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:23:34.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:34.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:34.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/logs
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.3:35001
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.3
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37591
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.3
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:34.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:23:34.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:34.465010 17005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:23:34.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:34.465752 17005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:23:34.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:34.466214 17005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:23:34.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:34.505386 17005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:23:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:34.506322 17005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.3
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:34.547991 17005 tablet_server_runner.cc:78] Tablet server non-default flags:
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.3:35001
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.3
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37591
05:23:34.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:34.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:23:34.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:34.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:23:34.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:23:34.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:34.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.3
05:23:34.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/logs
05:23:34.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:34.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:23:34.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:23:34.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:23:34.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:23:34.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:23:34.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:23:34.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:23:34.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:23:34.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:23:34.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:34.549324 17005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:23:34.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:34.551119 17005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:23:34.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:34.574021 17013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:35.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:34.574040 17014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:35.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:34.576000 17005 server_base.cc:1034] running on GCE node
05:23:35.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:34.574236 17016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:35.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.839946 17005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:23:35.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.843075 17005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:23:35.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.844606 17005 hybrid_clock.cc:648] HybridClock initialized: now 1783488215844568 us; error 55 us; skew 500 ppm
05:23:35.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.845403 17005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:23:35.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.855816 17005 webserver.cc:460] Webserver started at http://127.14.90.3:37591/ using document root <none> and password file <none>
05:23:35.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.856945 17005 fs_manager.cc:359] Metadata directory not provided
05:23:35.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.857194 17005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:23:35.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.865432 17005 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.000s
05:23:35.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.870982 17023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:23:35.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.872128 17005 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.872511 17005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:46 on dist-test-slave-zq0g"
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.875012 17005 fs_report.cc:383] FS layout report
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/wal
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/data
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:23:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:23:35.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:23:35.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:23:35.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:23:35.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:23:35.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:23:35.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.915679 17005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:23:35.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.917304 17005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:23:35.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.917824 17005 kserver.cc:163] Server-wide thread pool size limit: 3276
05:23:35.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.920619 17005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:23:35.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.927165 17030 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:23:35.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.935108 17005 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:23:35.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.935338 17005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.003s
05:23:35.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.935542 17005 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:23:35.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.940490 17005 ts_tablet_manager.cc:610] Registered 1 tablets
05:23:35.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.940766 17005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.003s	sys 0.001s
05:23:35.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:35.941288 17030 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Bootstrap starting.
05:23:36.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.019941 17030 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Log is configured to *not* fsync() on all Append() calls
05:23:36.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.160678 17005 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.3:35001
05:23:36.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.160923 17140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.3:35001 every 8 connection(s)
05:23:36.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.163655 17005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-2/data/info.pb
05:23:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.165537 17030 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:23:36.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.166846 17030 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Bootstrap complete.
05:23:36.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.169884 17030 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Time spent bootstrapping tablet: real 0.229s	user 0.177s	sys 0.042s
05:23:36.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.176121 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 17005
05:23:36.177 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:36.255 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:23:36.259 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.14.90.1:46843
05:23:36.260 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:23:36.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.260831 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 16647
05:23:36.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.283993 17141 heartbeater.cc:347] Connected to a master server at 127.14.90.60:44831
05:23:36.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.284581 17141 heartbeater.cc:464] Registering TS with master...
05:23:36.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.286453 17141 heartbeater.cc:511] Master 127.14.90.60:44831 requested a full tablet report, sending...
05:23:36.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.276232 17030 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:36.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.289218 17030 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db66d29aaa8408c8acbad8bde8e4ae3, State: Initialized, Role: FOLLOWER
05:23:36.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.290589 17030 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:36.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.302506 14894 ts_manager.cc:194] Re-registered known tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:36.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.304020 17142 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:23:36.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.304440 17142 heartbeater.cc:464] Registering TS with master...
05:23:36.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.305158 14894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.90.3:52881
05:23:36.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.305420 17142 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:23:36.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.308346 17030 ts_tablet_manager.cc:1428] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3: Time spent starting tablet: real 0.138s	user 0.035s	sys 0.000s
05:23:36.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.309408 14816 ts_manager.cc:194] Re-registered known tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:36.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.310022 17143 heartbeater.cc:347] Connected to a master server at 127.14.90.62:44495
05:23:36.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.310387 17143 heartbeater.cc:464] Registering TS with master...
05:23:36.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.311450 17143 heartbeater.cc:511] Master 127.14.90.62:44495 requested a full tablet report, sending...
05:23:36.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.312211 17141 heartbeater.cc:503] Master 127.14.90.60:44831 was elected leader, sending a full tablet report...
05:23:36.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:36.315565 14740 ts_manager.cc:194] Re-registered known tserver with Master: 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:36.327 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] handling channelInactive
05:23:36.328 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state READY due to: connection closed
05:23:36.328 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:36.329 [DEBUG - Time-limited test] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:36.330 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.330 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.330 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.354 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:36.354 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:36.355 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [26ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2074456461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3415e5c6 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42ef3ffa -> wakeup thread Time-limited test)) after lookup
05:23:36.355 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:36.357 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.357 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.357 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.373 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:36.373 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:36.374 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [26ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [27ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [29ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [29ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [45ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2074456461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3415e5c6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@495684163 after retry RPC@639661799), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42ef3ffa -> wakeup thread Time-limited test -> (continuation of Deferred@495684163 after retry RPC@639661799))) after lookup
05:23:36.374 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:36.375 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.375 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.375 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.394 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:36.395 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:36.396 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=67), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [26ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [27ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [29ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [29ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [45ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [46ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [47ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [47ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [66ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2074456461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3415e5c6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@495684163 after retry RPC@639661799) -> (continuation of Deferred@162528715 after retry RPC@551265872), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42ef3ffa -> wakeup thread Time-limited test -> (continuation of Deferred@495684163 after retry RPC@639661799) -> (continuation of Deferred@162528715 after retry RPC@551265872))) after lookup
05:23:36.396 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:36.397 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.398 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.398 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.475 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:36.475 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:36.476 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=147), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [26ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [27ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [29ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [29ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [45ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [46ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [47ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [47ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [66ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [68ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [70ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [70ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [142ms] refreshing cache from master, [143ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [146ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2074456461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3415e5c6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@495684163 after retry RPC@639661799) -> (continuation of Deferred@162528715 after retry RPC@551265872) -> (continuation of Deferred@1710150860 after retry RPC@410650375), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42ef3ffa -> wakeup thread Time-limited test -> (continuation of Deferred@495684163 after retry RPC@639661799) -> (continuation of Deferred@162528715 after retry RPC@551265872) -> (continuation of Deferred@1710150860 after retry RPC@410650375))) after lookup
05:23:36.477 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:36.478 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.478 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.478 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.793 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:36.794 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)[L],a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)]}]
05:23:36.794 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=466), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [26ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [27ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [29ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [29ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [45ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [46ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [47ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [47ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [66ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [68ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [70ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [70ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [142ms] refreshing cache from master, [143ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [146ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [148ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [150ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [150ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [462ms] refreshing cache from master, [462ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [465ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2074456461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3415e5c6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@495684163 after retry RPC@639661799) -> (continuation of Deferred@162528715 after retry RPC@551265872) -> (continuation of Deferred@1710150860 after retry RPC@410650375) -> (continuation of Deferred@875706183 after retry RPC@1309738354), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42ef3ffa -> wakeup thread Time-limited test -> (continuation of Deferred@495684163 after retry RPC@639661799) -> (continuation of Deferred@162528715 after retry RPC@551265872) -> (continuation of Deferred@1710150860 after retry RPC@410650375) -> (continuation of Deferred@875706183 after retry RPC@1309738354))) after lookup
05:23:36.795 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] connecting to peer
05:23:36.796 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.796 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] cleaning up while in state CONNECTING due to: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:36.797 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843) for tablet 02161a9b84f144a4b9f0c67986f3fc46: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843
05:23:37.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.596421 17153 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 969c70d1882541a4b320cca5f9979912)
05:23:37.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.597061 17153 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:37.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.599866 17153 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:37.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:37.603173 16849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:23:37.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:37.607019 16849 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:23:37.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.615671 17096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" is_pre_election: true
05:23:37.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.616794 17096 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f24784906a4e2ebb7a4d8073e17074 in term 9.
05:23:37.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.618623 16849 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, a2f24784906a4e2ebb7a4d8073e17074; no voters: 969c70d1882541a4b320cca5f9979912
05:23:37.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.619488 17153 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 10 FOLLOWER]: Leader pre-election won for term 11
05:23:37.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.620116 17153 raft_consensus.cc:482] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 969c70d1882541a4b320cca5f9979912)
05:23:37.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.620399 17153 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 10 FOLLOWER]: Advancing to term 11
05:23:37.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.629769 17153 raft_consensus.cc:504] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:37.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.632162 17153 leader_election.cc:288] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 11 election: Requested vote from peers 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843), 3db66d29aaa8408c8acbad8bde8e4ae3 (127.14.90.3:35001)
05:23:37.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.634986 17096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02161a9b84f144a4b9f0c67986f3fc46" candidate_uuid: "a2f24784906a4e2ebb7a4d8073e17074" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3"
05:23:37.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.635874 17096 raft_consensus.cc:3037] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 9 FOLLOWER]: Advancing to term 11
05:23:37.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:37.637302 16849 leader_election.cc:334] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111)
05:23:37.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.644236 17096 raft_consensus.cc:2454] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f24784906a4e2ebb7a4d8073e17074 in term 11.
05:23:37.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.645387 16849 leader_election.cc:302] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db66d29aaa8408c8acbad8bde8e4ae3, a2f24784906a4e2ebb7a4d8073e17074; no voters: 969c70d1882541a4b320cca5f9979912
05:23:37.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.646035 17153 raft_consensus.cc:2781] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 11 FOLLOWER]: Leader election won for term 11
05:23:37.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.647869 17153 raft_consensus.cc:686] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [term 11 LEADER]: Becoming Leader. State: Replica: a2f24784906a4e2ebb7a4d8073e17074, State: Running, Role: LEADER
05:23:37.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.648875 17153 consensus_queue.cc:237] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:37.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.659526 14894 catalog_manager.cc:5495] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 reported cstate change: term changed from 10 to 11, leader changed from 969c70d1882541a4b320cca5f9979912 (127.14.90.1) to a2f24784906a4e2ebb7a4d8073e17074 (127.14.90.2). New cstate: current_term: 11 leader_uuid: "a2f24784906a4e2ebb7a4d8073e17074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } health_report { overall_health: UNKNOWN } } }
05:23:37.834 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 02161a9b84f144a4b9f0c67986f3fc46 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696' with partition [<start>, <end>)
05:23:37.835 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=02161a9b84f144a4b9f0c67986f3fc46@[3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001),969c70d1882541a4b320cca5f9979912(127.14.90.1:46843),a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)[L]]}]
05:23:37.835 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=02161a9b84f144a4b9f0c67986f3fc46, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1506), Traces: [0ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [2ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [2ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [26ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [27ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [29ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [29ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [45ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [46ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [47ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [47ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [66ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [68ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [70ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [70ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [142ms] refreshing cache from master, [143ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [146ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [148ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [150ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [150ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [462ms] refreshing cache from master, [462ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [465ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, [467ms] sending RPC to server 969c70d1882541a4b320cca5f9979912, [469ms] received response from server 969c70d1882541a4b320cca5f9979912: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [469ms] delaying RPC due to: Network error: Failed to connect to peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843): Connection refused: /127.14.90.1:46843, [1502ms] refreshing cache from master, [1502ms] Sub RPC GetTableLocations: sending RPC to server master-127.14.90.60:44831, [1506ms] Sub RPC GetTableLocations: received response from server master-127.14.90.60:44831: OK, deferred=Deferred@2074456461(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/2007160576@3415e5c6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@495684163 after retry RPC@639661799) -> (continuation of Deferred@162528715 after retry RPC@551265872) -> (continuation of Deferred@1710150860 after retry RPC@410650375) -> (continuation of Deferred@875706183 after retry RPC@1309738354) -> (continuation of Deferred@911074032 after retry RPC@1227094922), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42ef3ffa -> wakeup thread Time-limited test -> (continuation of Deferred@495684163 after retry RPC@639661799) -> (continuation of Deferred@162528715 after retry RPC@551265872) -> (continuation of Deferred@1710150860 after retry RPC@410650375) -> (continuation of Deferred@875706183 after retry RPC@1309738354) -> (continuation of Deferred@911074032 after retry RPC@1227094922))) after lookup
05:23:37.836 [DEBUG - kudu-nio-1] (Connection.java:748) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] connecting to peer
05:23:37.837 [DEBUG - kudu-nio-1] (Connection.java:757) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] Successfully connected to peer
05:23:37.863 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xe83398e9, L:/127.0.0.1:41492 - R:/127.14.90.2:45685] using TOKEN/null
05:23:37.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.873217 17096 raft_consensus.cc:1261] T 02161a9b84f144a4b9f0c67986f3fc46 P 3db66d29aaa8408c8acbad8bde8e4ae3 [term 11 FOLLOWER]: Refusing update from remote peer a2f24784906a4e2ebb7a4d8073e17074: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 42. (index mismatch)
05:23:37.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:37.875262 16849 consensus_peers.cc:487] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 -> Peer 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843): Couldn't send request to peer 969c70d1882541a4b320cca5f9979912. Status: Network error: Client connection negotiation failed: client connection to 127.14.90.1:46843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:23:37.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.875416 17158 consensus_queue.cc:1035] T 02161a9b84f144a4b9f0c67986f3fc46 P a2f24784906a4e2ebb7a4d8073e17074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.001s
05:23:37.942 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.14.90.1:46843
05:23:37.943 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:23:37.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:37.944646 14696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:37.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu
05:23:37.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/logs
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:23:37.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:23:37.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.1:46843
05:23:37.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.1
05:23:37.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:37.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34663
05:23:37.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.1
05:23:37.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:37.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:37.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:37.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:23:38.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:38.342509 17181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:23:38.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:38.343111 17181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:23:38.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:38.343716 17181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:23:38.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:38.375852 17181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:23:38.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:38.376739 17181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.90.1
05:23:38.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:38.414265 17181 tablet_server_runner.cc:78] Tablet server non-default flags:
05:23:38.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:23:38.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.90.20:35971
05:23:38.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:23:38.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.90.1:46843
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.90.1
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34663
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.90.62:44495,127.14.90.61:34695,127.14.90.60:44831
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.90.1
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/logs
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 04:09:51 UTC on ccycloud.quasar-cakjni.root.comops.site
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492623
05:23:38.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
05:23:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:38.415897 17181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:23:38.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:38.417723 17181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:23:38.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:38.439960 17188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:39.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:38.440048 17190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:39.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:38.441898 17181 server_base.cc:1034] running on GCE node
05:23:39.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:38.440018 17187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:23:39.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.696520 17181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:23:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.698973 17181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:23:39.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.700429 17181 hybrid_clock.cc:648] HybridClock initialized: now 1783488219700338 us; error 55 us; skew 500 ppm
05:23:39.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.701488 17181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:23:39.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.708163 17181 webserver.cc:460] Webserver started at http://127.14.90.1:34663/ using document root <none> and password file <none>
05:23:39.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.709110 17181 fs_manager.cc:359] Metadata directory not provided
05:23:39.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.709309 17181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:23:39.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.717788 17181 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.002s	sys 0.004s
05:23:39.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.724006 17199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:23:39.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.725442 17181 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.000s	sys 0.005s
05:23:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.725844 17181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data,/tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "969c70d1882541a4b320cca5f9979912"
05:23:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:22:42 on dist-test-slave-zq0g"
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.728480 17181 fs_report.cc:383] FS layout report
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/wal
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/data
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:23:39.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:23:39.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.775846 17181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:23:39.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.777515 17181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:23:39.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.778088 17181 kserver.cc:163] Server-wide thread pool size limit: 3276
05:23:39.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.780984 17181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:23:39.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.787856 17206 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:23:39.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.796669 17181 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:23:39.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.797012 17181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
05:23:39.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.797366 17181 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:23:39.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.802793 17181 ts_tablet_manager.cc:610] Registered 1 tablets
05:23:39.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.803000 17181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.005s	sys 0.001s
05:23:39.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.803488 17206 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap starting.
05:23:39.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:39.881181 17206 log.cc:824] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Log is configured to *not* fsync() on all Append() calls
05:23:40.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.007447 17181 rpc_server.cc:307] RPC server started. Bound to: 127.14.90.1:46843
05:23:40.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.007668 17313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.90.1:46843 every 8 connection(s)
05:23:40.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.010197 17181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskarRtQd/test-tmp/mini-kudu-cluster4977987233834906985/ts-0/data/info.pb
05:23:40.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.019735 14696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu as pid 17181
05:23:40.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.020376 17206 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:23:40.021 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:23:40.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.021521 17206 tablet_bootstrap.cc:492] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Bootstrap complete.
05:23:40.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.023582 17206 ts_tablet_manager.cc:1397] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912: Time spent bootstrapping tablet: real 0.221s	user 0.158s	sys 0.056s
05:23:40.059 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:23:40.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.062937 17206 raft_consensus.cc:348] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:40.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.076313 17206 raft_consensus.cc:729] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969c70d1882541a4b320cca5f9979912, State: Initialized, Role: FOLLOWER
05:23:40.078 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783488167696, tablet=02161a9b84f144a4b9f0c67986f3fc46, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:23:40.086 [DEBUG - main] (Connection.java:575) [peer master-127.14.90.62:44495(127.14.90.62:44495)] disconnecting while in state READY
05:23:40.090 [DEBUG - main] (Connection.java:575) [peer 3db66d29aaa8408c8acbad8bde8e4ae3(127.14.90.3:35001)] disconnecting while in state TERMINATED
05:23:40.091 [DEBUG - main] (Connection.java:575) [peer 969c70d1882541a4b320cca5f9979912(127.14.90.1:46843)] disconnecting while in state TERMINATED
05:23:40.091 [DEBUG - main] (Connection.java:575) [peer master-127.14.90.60:44831(127.14.90.60:44831)] disconnecting while in state READY
05:23:40.091 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.14.90.62:44495(127.14.90.62:44495)] handling channelInactive
05:23:40.092 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.14.90.62:44495(127.14.90.62:44495)] cleaning up while in state READY due to: connection closed
05:23:40.092 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.14.90.60:44831(127.14.90.60:44831)] handling channelInactive
05:23:40.093 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.14.90.60:44831(127.14.90.60:44831)] cleaning up while in state READY due to: connection closed
05:23:40.093 [DEBUG - main] (Connection.java:575) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] disconnecting while in state READY
05:23:40.093 [DEBUG - main] (Connection.java:575) [peer master-127.14.90.61:34695(127.14.90.61:34695)] disconnecting while in state READY
05:23:40.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.087105 17315 heartbeater.cc:347] Connected to a master server at 127.14.90.61:34695
05:23:40.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.096158 17315 heartbeater.cc:464] Registering TS with master...
05:23:40.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.097890 17315 heartbeater.cc:511] Master 127.14.90.61:34695 requested a full tablet report, sending...
05:23:40.098 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.14.90.61:34695(127.14.90.61:34695)] handling channelInactive
05:23:40.098 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.14.90.61:34695(127.14.90.61:34695)] cleaning up while in state READY due to: connection closed
05:23:40.100 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2704) Releasing all remaining resources
05:23:40.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.100571 17206 consensus_queue.cc:260] T 02161a9b84f144a4b9f0c67986f3fc46 P 969c70d1882541a4b320cca5f9979912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969c70d1882541a4b320cca5f9979912" member_type: VOTER last_known_addr { host: "127.14.90.1" port: 46843 } } peers { permanent_uuid: "a2f24784906a4e2ebb7a4d8073e17074" member_type: VOTER last_known_addr { host: "127.14.90.2" port: 45685 } } peers { permanent_uuid: "3db66d29aaa8408c8acbad8bde8e4ae3" member_type: VOTER last_known_addr { host: "127.14.90.3" port: 35001 } }
05:23:40.103 [DEBUG - kudu-nio-1] (Connection.java:239) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] handling channelInactive
05:23:40.103 [DEBUG - kudu-nio-1] (Connection.java:709) [peer a2f24784906a4e2ebb7a4d8073e17074(127.14.90.2:45685)] cleaning up while in state READY due to: connection closed
05:23:40.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.111780 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 17181
05:23:40.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.113572 14815 ts_manager.cc:194] Re-registered known tserver with Master: 969c70d1882541a4b320cca5f9979912 (127.14.90.1:46843)
05:23:40.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:40.151734 14803 connection.cc:535] server connection from 127.14.90.1:56737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
05:23:40.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.152356 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 16826
05:23:40.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:40.155093 17319 negotiation.cc:337] Failed RPC negotiation. Trace:
05:23:40.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.020814 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.90.1:33475 (local address 127.14.90.60:44831)
05:23:40.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.048426 (+ 27612us) server_negotiation.cc:206] Beginning negotiation
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.048462 (+    36us) server_negotiation.cc:399] Waiting for connection header
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.048533 (+    71us) server_negotiation.cc:407] Connection header received
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.069116 (+ 20583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.069152 (+    36us) server_negotiation.cc:461] Received NEGOTIATE request from client
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.069481 (+   329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.069748 (+   267us) server_negotiation.cc:226] Negotiated authn=SASL
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.071996 (+  2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.074289 (+  2293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.088485 (+ 14196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.089670 (+  1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.089977 (+   307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.152228 (+ 62251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.90.1:33475: BlockingRecv error: recv got EOF from 127.14.90.1:33475 (error 108)
05:23:40.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":22751,"thread_start_us":22368,"threads_started":1}
05:23:40.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:23:40.157543 17323 negotiation.cc:337] Failed RPC negotiation. Trace:
05:23:40.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.055860 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.90.1:50495 (local address 127.14.90.62:44495)
05:23:40.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.116112 (+ 60252us) server_negotiation.cc:206] Beginning negotiation
05:23:40.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.116154 (+    42us) server_negotiation.cc:399] Waiting for connection header
05:23:40.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.116214 (+    60us) server_negotiation.cc:407] Connection header received
05:23:40.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.116374 (+   160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
05:23:40.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.116401 (+    27us) server_negotiation.cc:461] Received NEGOTIATE request from client
05:23:40.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.116706 (+   305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
05:23:40.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.116969 (+   263us) server_negotiation.cc:226] Negotiated authn=SASL
05:23:40.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:23:40.155734 (+ 38765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.90.1:50495: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
05:23:40.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":59922,"thread_start_us":59647,"threads_started":1}
05:23:40.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.187763 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 17005
05:23:40.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.222416 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 14708
05:23:40.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.255525 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 14778
05:23:40.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:23:40.288079 14696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskarRtQd/build/tsan/bin/kudu with pid 14853
05:23:41.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T05:23:41Z chronyd exiting

Time: 70.147

OK (1 test)

