JUnit version 4.13.2
.14:21:54.417 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1779114114407
14:21:54.472 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
14:21:54.505 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
14:21:54.517 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskq0uqLp/test-tmp
14:21:54.547 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
14:21:54.550 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskq0uqLp/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
14:21:54.790 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

14:21:54.992 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:21:54.993 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

14:21:54.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:54.997534 26474 mini_oidc.cc:150] Starting JWKS server
14:21:55.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:55.000695 26474 webserver.cc:533] Webserver started at http://127.0.0.1:41015/ using document root <none> and password file <none>
14:21:55.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:55.001106 26474 mini_oidc.cc:186] Starting OIDC Discovery server
14:21:55.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:55.002856 26474 webserver.cc:533] Webserver started at http://127.0.0.1:34193/ using document root <none> and password file <none>
14:21:55.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T14:21:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
14:21:55.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T14:21:55Z Disabled control of system clock
14:21:55.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:55.067611 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:21:55.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:21:55.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/wal
14:21:55.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/data
14:21:55.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:21:55.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:21:55.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:21:55.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:21:55.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:21:55.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:21:55.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/logs
14:21:55.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/data/info.pb
14:21:55.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:21:55.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:21:55.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:21:55.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:21:55.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:21:55.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:21:55.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:21:55.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:21:55.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:21:55.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:21:55.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:21:55.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:21:55.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.190:42765
14:21:55.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.190
14:21:55.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:21:55.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:21:55.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:21:55.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:21:55.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:21:55.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:21:55.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:21:55.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:21:55.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41015/jwks.json with env {}
14:21:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:55.374070 26486 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:21:55.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:55.374636 26486 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:21:55.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:55.375077 26486 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:21:55.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:55.408743 26486 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:21:55.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:55.409118 26486 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:21:55.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:55.409420 26486 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:21:55.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:55.409650 26486 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:21:55.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:55.409933 26486 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:21:55.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:55.410092 26486 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:41015/jwks.json
14:21:55.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:55.447954 26486 master_runner.cc:387] Master server non-default flags:
14:21:55.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:21:55.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:21:55.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:21:55.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:21:55.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/data
14:21:55.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/wal
14:21:55.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:21:55.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:21:55.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:21:55.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:21:55.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:21:55.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.190:42765
14:21:55.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:21:55.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:21:55.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:21:55.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41015/jwks.json
14:21:55.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:21:55.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:21:55.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/data/info.pb
14:21:55.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.190
14:21:55.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:21:55.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:21:55.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:21:55.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:21:55.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:21:55.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:21:55.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/logs
14:21:55.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:21:55.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:21:55.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:21:55.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:21:55.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:21:55.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:21:55.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:21:55.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:21:55.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:21:55.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:21:55.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:55.449358 26486 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:21:55.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:55.451069 26486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:21:55.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:55.463374 26492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:21:56.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:55.464627 26493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:21:56.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:56.707881 26495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:21:56.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:56.710935 26494 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1242 milliseconds
14:21:56.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.711035 26486 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:21:56.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.712409 26486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:21:56.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.715546 26486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:21:56.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.717038 26486 hybrid_clock.cc:648] HybridClock initialized: now 1779114116716989 us; error 61 us; skew 500 ppm
14:21:56.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.717859 26486 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:21:56.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.725051 26486 webserver.cc:533] Webserver started at http://127.25.218.190:36517/ using document root <none> and password file <none>
14:21:56.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.726079 26486 fs_manager.cc:362] Metadata directory not provided
14:21:56.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.726274 26486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:21:56.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.726704 26486 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:21:56.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.731405 26486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/data/instance:
14:21:56.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5785468a3d744ab9b274e7ce2112a2a"
14:21:56.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:21:56 on dist-test-slave-45s3"
14:21:56.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.732611 26486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/wal/instance:
14:21:56.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5785468a3d744ab9b274e7ce2112a2a"
14:21:56.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:21:56 on dist-test-slave-45s3"
14:21:56.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.740345 26486 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.003s	sys 0.005s
14:21:56.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.746239 26502 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:21:56.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.747529 26486 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
14:21:56.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.747947 26486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/wal
14:21:56.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5785468a3d744ab9b274e7ce2112a2a"
14:21:56.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:21:56 on dist-test-slave-45s3"
14:21:56.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.748420 26486 fs_report.cc:389] FS layout report
14:21:56.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:21:56.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/wal
14:21:56.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/wal
14:21:56.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/data/data
14:21:56.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:21:56.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:21:56.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:21:56.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:21:56.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:21:56.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:21:56.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:21:56.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:21:56.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:21:56.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:21:56.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:21:56.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.812718 26486 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:21:56.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.814515 26486 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:21:56.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.815063 26486 kserver.cc:163] Server-wide thread pool size limit: 3276
14:21:56.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.892712 26486 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.190:42765
14:21:56.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.892764 26553 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.190:42765 every 8 connection(s)
14:21:56.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.896203 26486 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/data/info.pb
14:21:56.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.901707 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 26486
14:21:56.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.902276 26474 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-0/wal/instance
14:21:56.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.904129 26554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:21:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.907569 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:21:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:21:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/wal
14:21:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/data
14:21:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:21:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:21:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:21:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:21:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:21:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:21:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/logs
14:21:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/data/info.pb
14:21:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:21:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:21:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:21:56.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:21:56.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:21:56.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:21:56.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:21:56.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:21:56.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:21:56.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:21:56.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:21:56.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:21:56.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.189:41079
14:21:56.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.189
14:21:56.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:21:56.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:21:56.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:21:56.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:21:56.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:21:56.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:21:56.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:21:56.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:21:56.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41015/jwks.json with env {}
14:21:56.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.918556 26554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } has no permanent_uuid. Determining permanent_uuid...
14:21:56.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:56.953987 26554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } has no permanent_uuid. Determining permanent_uuid...
14:21:56.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:56.956652 26503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.189:41079: connect: Connection refused (error 111)
14:21:56.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:56.960817 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.189:41079: Network error: Client connection negotiation failed: client connection to 127.25.218.189:41079: connect: Connection refused (error 111)
14:21:57.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:57.010380 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } attempt: 1
14:21:57.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.014628 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.189:41079: Network error: Client connection negotiation failed: client connection to 127.25.218.189:41079: connect: Connection refused (error 111)
14:21:57.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:57.083010 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } attempt: 2
14:21:57.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.087342 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.189:41079: Network error: Client connection negotiation failed: client connection to 127.25.218.189:41079: connect: Connection refused (error 111)
14:21:57.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:57.178853 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } attempt: 3
14:21:57.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.183202 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.189:41079: Network error: Client connection negotiation failed: client connection to 127.25.218.189:41079: connect: Connection refused (error 111)
14:21:57.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.234479 26556 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:21:57.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.235050 26556 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:21:57.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.235502 26556 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:21:57.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.268401 26556 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:21:57.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.268690 26556 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:21:57.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.268884 26556 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:21:57.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.269079 26556 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:21:57.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.269263 26556 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:21:57.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.269481 26556 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:41015/jwks.json
14:21:57.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:57.307999 26556 master_runner.cc:387] Master server non-default flags:
14:21:57.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:21:57.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:21:57.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:21:57.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:21:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/data
14:21:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/wal
14:21:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:21:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:21:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:21:57.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:21:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:21:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.189:41079
14:21:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:21:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:21:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:21:57.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41015/jwks.json
14:21:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:21:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:21:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/data/info.pb
14:21:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.189
14:21:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:21:57.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:21:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:21:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:21:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:21:57.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:21:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/logs
14:21:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:21:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:21:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:21:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:21:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:21:57.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:21:57.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:21:57.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:21:57.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:21:57.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:21:57.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:57.309854 26556 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:21:57.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:57.312331 26556 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:21:57.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.325857 26565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:21:57.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:57.326673 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } attempt: 4
14:21:57.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.332100 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.189:41079: Network error: Client connection negotiation failed: client connection to 127.25.218.189:41079: connect: Connection refused (error 111)
14:21:57.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:57.631632 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } attempt: 5
14:21:57.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.638856 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.189:41079: Network error: Client connection negotiation failed: client connection to 127.25.218.189:41079: connect: Connection refused (error 111)
14:21:58.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.186486 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } attempt: 6
14:21:58.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:58.201438 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.189:41079: Network error: Client connection negotiation failed: client connection to 127.25.218.189:41079: connect: Connection refused (error 111)
14:21:58.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.330755 26568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:21:58.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:57.327664 26566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:21:58.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:58.668084 26567 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1334 milliseconds
14:21:58.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.668252 26556 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:21:58.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.670047 26556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:21:58.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.673910 26556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:21:58.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.675443 26556 hybrid_clock.cc:648] HybridClock initialized: now 1779114118675436 us; error 91 us; skew 500 ppm
14:21:58.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.676831 26556 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:21:58.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.684715 26556 webserver.cc:533] Webserver started at http://127.25.218.189:41613/ using document root <none> and password file <none>
14:21:58.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.686076 26556 fs_manager.cc:362] Metadata directory not provided
14:21:58.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.686388 26556 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:21:58.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.687182 26556 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:21:58.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.692662 26556 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/data/instance:
14:21:58.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d02d61c00b7148d1964b578087cb7480"
14:21:58.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:21:58 on dist-test-slave-45s3"
14:21:58.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.694357 26556 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/wal/instance:
14:21:58.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d02d61c00b7148d1964b578087cb7480"
14:21:58.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:21:58 on dist-test-slave-45s3"
14:21:58.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.703493 26556 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.010s	sys 0.001s
14:21:58.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.710470 26577 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:21:58.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.711934 26556 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.007s	sys 0.000s
14:21:58.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.712388 26556 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/wal
14:21:58.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d02d61c00b7148d1964b578087cb7480"
14:21:58.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:21:58 on dist-test-slave-45s3"
14:21:58.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.712888 26556 fs_report.cc:389] FS layout report
14:21:58.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:21:58.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/wal
14:21:58.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/wal
14:21:58.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/data/data
14:21:58.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:21:58.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:21:58.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:21:58.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:21:58.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:21:58.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:21:58.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:21:58.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:21:58.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:21:58.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:21:58.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:21:58.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.778168 26556 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:21:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.780118 26556 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:21:58.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.780720 26556 kserver.cc:163] Server-wide thread pool size limit: 3276
14:21:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.861331 26556 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.189:41079
14:21:58.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.861450 26628 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.189:41079 every 8 connection(s)
14:21:58.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.864916 26556 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/data/info.pb
14:21:58.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.872355 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 26556
14:21:58.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.872836 26474 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-1/wal/instance
14:21:58.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.872867 26629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:21:58.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.875301 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:21:58.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:21:58.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/wal
14:21:58.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/data
14:21:58.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:21:58.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:21:58.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:21:58.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:21:58.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:21:58.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:21:58.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/logs
14:21:58.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/data/info.pb
14:21:58.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:21:58.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:21:58.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:21:58.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:21:58.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:21:58.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:21:58.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:21:58.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:21:58.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:21:58.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:21:58.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:21:58.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:21:58.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.188:34713
14:21:58.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.188
14:21:58.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:21:58.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:21:58.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:21:58.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:21:58.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:21:58.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:21:58.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:21:58.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:21:58.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41015/jwks.json with env {}
14:21:58.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.889137 26629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } has no permanent_uuid. Determining permanent_uuid...
14:21:58.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.913295 26629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } has no permanent_uuid. Determining permanent_uuid...
14:21:58.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.928179 26629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } has no permanent_uuid. Determining permanent_uuid...
14:21:58.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:58.930788 26578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:58.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:58.934928 26629 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:58.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:58.984508 26629 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 1
14:21:58.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:58.989118 26629 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:59.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.057576 26629 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 2
14:21:59.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.061754 26629 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:59.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.153209 26629 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 3
14:21:59.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.157507 26629 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:59.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.216497 26631 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:21:59.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.217049 26631 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:21:59.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.217661 26631 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:21:59.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.250468 26631 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:21:59.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.250752 26631 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:21:59.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.250943 26631 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:21:59.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.251138 26631 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:21:59.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.251322 26631 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:21:59.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.251549 26631 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:41015/jwks.json
14:21:59.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.262064 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } attempt: 7
14:21:59.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.274075 26554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } has no permanent_uuid. Determining permanent_uuid...
14:21:59.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.279579 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:59.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.296658 26631 master_runner.cc:387] Master server non-default flags:
14:21:59.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:21:59.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:21:59.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:21:59.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:21:59.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/data
14:21:59.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/wal
14:21:59.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:21:59.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:21:59.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:21:59.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:21:59.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:21:59.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.188:34713
14:21:59.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:21:59.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:21:59.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:21:59.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41015/jwks.json
14:21:59.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:21:59.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:21:59.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/data/info.pb
14:21:59.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.188
14:21:59.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:21:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:21:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:21:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:21:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:21:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:21:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/logs
14:21:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:21:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:21:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:21:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:21:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:21:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:21:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:21:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:21:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:21:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:21:59.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.298007 26631 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:21:59.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.299751 26631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:21:59.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.300959 26629 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 4
14:21:59.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.305858 26629 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:59.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.313517 26643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:21:59.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.338073 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 1
14:21:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.345992 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:59.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.427619 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 2
14:21:59.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.435488 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:59.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.521085 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 3
14:21:59.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.529995 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:59.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.605304 26629 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 5
14:21:59.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.621474 26629 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:59.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.670565 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 4
14:21:59.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.678407 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:21:59.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:21:59.961989 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 5
14:21:59.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.971194 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:22:00.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.169056 26629 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 6
14:22:00.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:00.200006 26629 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:22:00.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.523808 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 6
14:22:00.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:00.546447 26554 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.25.218.188:34713: Network error: Client connection negotiation failed: client connection to 127.25.218.188:34713: connect: Connection refused (error 111)
14:22:00.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:00.717403 26642 debug-util.cc:398] Leaking SignalData structure 0x7b08000271e0 after lost signal to thread 26631
14:22:00.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:00.949906 26642 kernel_stack_watchdog.cc:198] Thread 26631 stuck at /home/jenkins-slave/workspace/build_and_test@2/src/kudu/util/thread.cc:642 for 401ms:
14:22:00.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:22:00.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:22:00.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:22:00.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:22:00.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:21:59.314034 26644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:00.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:00.951593 26645 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1635 milliseconds
14:22:00.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:00.951604 26631 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.638s	user 0.001s	sys 0.002s
14:22:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:00.952788 26631 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.639s	user 0.001s	sys 0.002s
14:22:00.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:00.953662 26646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:00.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.953766 26631 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:22:00.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.957499 26631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:00.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.960858 26631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:00.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.962242 26631 hybrid_clock.cc:648] HybridClock initialized: now 1779114120962196 us; error 53 us; skew 500 ppm
14:22:00.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.963135 26631 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:00.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.970651 26631 webserver.cc:533] Webserver started at http://127.25.218.188:42581/ using document root <none> and password file <none>
14:22:00.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.971597 26631 fs_manager.cc:362] Metadata directory not provided
14:22:00.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.971896 26631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:00.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.972337 26631 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:22:00.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.977126 26631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/data/instance:
14:22:00.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec152e0ed90d4ed3949c35ab3b394da4"
14:22:00.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:00 on dist-test-slave-45s3"
14:22:00.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.978467 26631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/wal/instance:
14:22:00.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec152e0ed90d4ed3949c35ab3b394da4"
14:22:00.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:00 on dist-test-slave-45s3"
14:22:00.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.986761 26631 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.009s	sys 0.000s
14:22:00.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.993012 26657 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:00.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.994243 26631 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
14:22:00.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.994673 26631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/wal
14:22:00.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec152e0ed90d4ed3949c35ab3b394da4"
14:22:00.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:00 on dist-test-slave-45s3"
14:22:00.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:00.995103 26631 fs_report.cc:389] FS layout report
14:22:00.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:00.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/wal
14:22:00.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/wal
14:22:00.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/data/data
14:22:00.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:00.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:00.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:00.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:00.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:00.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:00.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:00.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:00.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:00.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:00.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:01.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.045181 26631 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:01.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.046964 26631 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:01.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.047529 26631 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:01.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.129522 26631 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.188:34713
14:22:01.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.129614 26708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.188:34713 every 8 connection(s)
14:22:01.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.132761 26631 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/data/info.pb
14:22:01.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.134331 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 26631
14:22:01.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.134910 26474 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/master-2/wal/instance
14:22:01.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.141269 26709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:22:01.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.161854 26709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } has no permanent_uuid. Determining permanent_uuid...
14:22:01.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.186817 26709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } has no permanent_uuid. Determining permanent_uuid...
14:22:01.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.197625 26709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } has no permanent_uuid. Determining permanent_uuid...
14:22:01.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.214762 26709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4: Bootstrap starting.
14:22:01.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.222699 26709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4: Neither blocks nor log segments found. Creating new log.
14:22:01.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.224723 26709 log.cc:826] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4: Log is configured to *not* fsync() on all Append() calls
14:22:01.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.229302 26709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4: No bootstrap required, opened a new log
14:22:01.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.255059 26709 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } }
14:22:01.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.255818 26709 raft_consensus.cc:385] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:22:01.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.256105 26709 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec152e0ed90d4ed3949c35ab3b394da4, State: Initialized, Role: FOLLOWER
14:22:01.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.256950 26709 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } }
14:22:01.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.259377 26718 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } } }
14:22:01.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.260059 26718 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [sys.catalog]: This master's current role is: FOLLOWER
14:22:01.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.260650 26629 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 7
14:22:01.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.260972 26709 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [sys.catalog]: configured and running, proceeding with master startup.
14:22:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.288216 26629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480: Bootstrap starting.
14:22:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:01.294559 26730 catalog_manager.cc:1594] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:22:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:01.295415 26730 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:22:01.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.296036 26629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480: Neither blocks nor log segments found. Creating new log.
14:22:01.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.298488 26629 log.cc:826] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480: Log is configured to *not* fsync() on all Append() calls
14:22:01.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:01.303205 26663 tablet.cc:2404] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:22:01.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.306346 26629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480: No bootstrap required, opened a new log
14:22:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.289585 26631 master_runner.cc:187] Error getting master registration for 127.25.218.190:42765: OK, instance_id { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" instance_seqno: 1779114116872421 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:22:01.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.323212 26631 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:22:01.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.327800 26629 raft_consensus.cc:359] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } }
14:22:01.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.328532 26629 raft_consensus.cc:385] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:22:01.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.328821 26629 raft_consensus.cc:740] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02d61c00b7148d1964b578087cb7480, State: Initialized, Role: FOLLOWER
14:22:01.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.329874 26629 consensus_queue.cc:260] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } }
14:22:01.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.332827 26733 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } } }
14:22:01.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.333688 26733 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [sys.catalog]: This master's current role is: FOLLOWER
14:22:01.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.334852 26629 sys_catalog.cc:565] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [sys.catalog]: configured and running, proceeding with master startup.
14:22:01.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:01.370371 26745 catalog_manager.cc:1594] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:22:01.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:01.371320 26745 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:22:01.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.365710 26556 master_runner.cc:187] Error getting master registration for 127.25.218.190:42765: OK, instance_id { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" instance_seqno: 1779114116872421 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:22:01.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.392843 26556 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:22:01.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.517660 26718 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:22:01.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.518297 26718 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } }
14:22:01.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.520992 26718 leader_election.cc:290] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5785468a3d744ab9b274e7ce2112a2a (127.25.218.190:42765), d02d61c00b7148d1964b578087cb7480 (127.25.218.189:41079)
14:22:01.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.522655 26604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02d61c00b7148d1964b578087cb7480" is_pre_election: true
14:22:01.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.523636 26604 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec152e0ed90d4ed3949c35ab3b394da4 in term 0.
14:22:01.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.525327 26658 leader_election.cc:304] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d02d61c00b7148d1964b578087cb7480, ec152e0ed90d4ed3949c35ab3b394da4; no voters: 
14:22:01.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.526271 26718 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
14:22:01.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.526602 26718 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:22:01.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.526909 26718 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [term 0 FOLLOWER]: Advancing to term 1
14:22:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.534509 26718 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } }
14:22:01.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.538376 26604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02d61c00b7148d1964b578087cb7480"
14:22:01.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.538985 26604 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [term 0 FOLLOWER]: Advancing to term 1
14:22:01.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:01.540076 26583 tablet.cc:2404] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:22:01.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.546741 26604 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec152e0ed90d4ed3949c35ab3b394da4 in term 1.
14:22:01.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.551432 26718 leader_election.cc:290] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [CANDIDATE]: Term 1 election: Requested vote from peers b5785468a3d744ab9b274e7ce2112a2a (127.25.218.190:42765), d02d61c00b7148d1964b578087cb7480 (127.25.218.189:41079)
14:22:01.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.556166 26658 leader_election.cc:304] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d02d61c00b7148d1964b578087cb7480, ec152e0ed90d4ed3949c35ab3b394da4; no voters: 
14:22:01.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.557250 26718 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [term 1 FOLLOWER]: Leader election won for term 1
14:22:01.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.561496 26718 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [term 1 LEADER]: Becoming Leader. State: Replica: ec152e0ed90d4ed3949c35ab3b394da4, State: Running, Role: LEADER
14:22:01.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.563673 26718 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } }
14:22:01.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.574967 26747 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec152e0ed90d4ed3949c35ab3b394da4. Latest consensus state: current_term: 1 leader_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } } }
14:22:01.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.575811 26747 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [sys.catalog]: This master's current role is: LEADER
14:22:01.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.578995 26749 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
14:22:01.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.579983 26554 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } attempt: 7
14:22:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.588994 26749 catalog_manager.cc:1520] Initializing Kudu cluster ID...
14:22:01.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.612519 26604 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [term 1 FOLLOWER]: Refusing update from remote peer ec152e0ed90d4ed3949c35ab3b394da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:22:01.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.614146 26747 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:22:01.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.617918 26554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a: Bootstrap starting.
14:22:01.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.638087 26554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a: Neither blocks nor log segments found. Creating new log.
14:22:01.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.641326 26554 log.cc:826] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a: Log is configured to *not* fsync() on all Append() calls
14:22:01.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.651266 26733 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec152e0ed90d4ed3949c35ab3b394da4. Latest consensus state: current_term: 1 leader_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } } }
14:22:01.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.652092 26733 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [sys.catalog]: This master's current role is: FOLLOWER
14:22:01.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.660534 26751 mvcc.cc:204] Tried to move back new op lower bound from 7287251442103394304 to 7287251441935069184. Current Snapshot: MvccSnapshot[applied={T|T < 7287251442103394304}]
14:22:01.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.664847 26718 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } } }
14:22:01.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.665653 26718 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [sys.catalog]: This master's current role is: LEADER
14:22:01.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.665405 26554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a: No bootstrap required, opened a new log
14:22:01.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.668612 26749 catalog_manager.cc:1383] Generated new cluster ID: e623ab0d02584d5091358c2b5e4dcf63
14:22:01.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.668934 26749 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
14:22:01.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.682561 26733 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } } }
14:22:01.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.683307 26733 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480 [sys.catalog]: This master's current role is: FOLLOWER
14:22:01.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.717187 26554 raft_consensus.cc:359] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } }
14:22:01.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.718159 26554 raft_consensus.cc:385] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:22:01.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.718448 26554 raft_consensus.cc:740] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5785468a3d744ab9b274e7ce2112a2a, State: Initialized, Role: FOLLOWER
14:22:01.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.719796 26554 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } }
14:22:01.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.724979 26554 sys_catalog.cc:565] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [sys.catalog]: configured and running, proceeding with master startup.
14:22:01.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.725452 26527 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [term 0 FOLLOWER]: Advancing to term 1
14:22:01.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.731166 26528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5785468a3d744ab9b274e7ce2112a2a"
14:22:01.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.732651 26529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5785468a3d744ab9b274e7ce2112a2a" is_pre_election: true
14:22:01.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.738350 26527 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [term 1 FOLLOWER]: Refusing update from remote peer ec152e0ed90d4ed3949c35ab3b394da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:22:01.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.724059 26757 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } } }
14:22:01.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.740373 26747 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:22:01.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.741346 26757 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [sys.catalog]: This master's current role is: FOLLOWER
14:22:01.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.767267 26749 catalog_manager.cc:1406] Generated new certificate authority record
14:22:01.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.769747 26749 catalog_manager.cc:1540] Loading token signing keys...
14:22:01.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.809046 26760 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } } }
14:22:01.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.809890 26760 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [sys.catalog]: This master's current role is: FOLLOWER
14:22:01.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.811121 26757 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec152e0ed90d4ed3949c35ab3b394da4. Latest consensus state: current_term: 1 leader_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } } }
14:22:01.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.812069 26757 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a [sys.catalog]: This master's current role is: FOLLOWER
14:22:01.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.819173 26749 catalog_manager.cc:6070] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4: Generated new TSK 0
14:22:01.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.820335 26749 catalog_manager.cc:1550] Initializing in-progress tserver states...
14:22:01.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.832670 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:01.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:01.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:01.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data
14:22:01.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:01.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:01.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:01.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:01.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:01.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:01.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/logs
14:22:01.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:01.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:01.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:01.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:01.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:01.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:01.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:01.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:01.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:01.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:01.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:01.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.129:0
14:22:01.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.129
14:22:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.129
14:22:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:22:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.832707 26718 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5785468a3d744ab9b274e7ce2112a2a" member_type: VOTER last_known_addr { host: "127.25.218.190" port: 42765 } } peers { permanent_uuid: "d02d61c00b7148d1964b578087cb7480" member_type: VOTER last_known_addr { host: "127.25.218.189" port: 41079 } } peers { permanent_uuid: "ec152e0ed90d4ed3949c35ab3b394da4" member_type: VOTER last_known_addr { host: "127.25.218.188" port: 34713 } } }
14:22:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.833454 26718 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec152e0ed90d4ed3949c35ab3b394da4 [sys.catalog]: This master's current role is: LEADER
14:22:01.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.952019 26775 catalog_manager.cc:1295] Loaded cluster ID: e623ab0d02584d5091358c2b5e4dcf63
14:22:01.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.952322 26775 catalog_manager.cc:1588] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a: loading cluster ID for follower catalog manager: success
14:22:01.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:01.975589 26775 catalog_manager.cc:1610] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a: acquiring CA information for follower catalog manager: success
14:22:02.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:02.005529 26775 catalog_manager.cc:1638] T 00000000000000000000000000000000 P b5785468a3d744ab9b274e7ce2112a2a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:22:02.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:02.377228 26745 catalog_manager.cc:1295] Loaded cluster ID: e623ab0d02584d5091358c2b5e4dcf63
14:22:02.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:02.378052 26745 catalog_manager.cc:1588] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480: loading cluster ID for follower catalog manager: success
14:22:02.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:02.385351 26745 catalog_manager.cc:1610] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480: acquiring CA information for follower catalog manager: success
14:22:02.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:02.396348 26745 catalog_manager.cc:1638] T 00000000000000000000000000000000 P d02d61c00b7148d1964b578087cb7480: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:22:02.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:02.403404 26772 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:02.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:02.404639 26772 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:02.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:02.405745 26772 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:02.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:02.468873 26772 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:02.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:02.470813 26772 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.129
14:22:02.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:02.542250 26772 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data
14:22:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.129:0
14:22:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.129
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.129
14:22:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/logs
14:22:02.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:02.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:02.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:02.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:02.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:02.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:02.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:02.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:02.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:02.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:02.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:02.554349 26772 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:02.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:02.557448 26772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:02.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:02.608608 26782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:05.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:05.036801 26772 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 2.428s	user 0.808s	sys 1.297s
14:22:05.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:05.039067 26772 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 2.431s	user 0.808s	sys 1.298s
14:22:05.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:03.988903 26781 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 26772
14:22:05.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:02.611320 26783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:05.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:05.049976 26788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:05.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.058485 26772 server_base.cc:1061] running on GCE node
14:22:05.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.060627 26772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:05.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.072259 26772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:05.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.074252 26772 hybrid_clock.cc:648] HybridClock initialized: now 1779114125074203 us; error 52 us; skew 500 ppm
14:22:05.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.075682 26772 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:05.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.101104 26772 webserver.cc:533] Webserver started at http://127.25.218.129:40225/ using document root <none> and password file <none>
14:22:05.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.102977 26772 fs_manager.cc:362] Metadata directory not provided
14:22:05.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.103607 26772 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:05.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.104439 26772 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:22:05.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.112337 26772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/instance:
14:22:05.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b575acd823514305a1d3e6f006cb011b"
14:22:05.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:05 on dist-test-slave-45s3"
14:22:05.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.114557 26772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal/instance:
14:22:05.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b575acd823514305a1d3e6f006cb011b"
14:22:05.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:05 on dist-test-slave-45s3"
14:22:05.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.130630 26772 fs_manager.cc:696] Time spent creating directory manager: real 0.015s	user 0.007s	sys 0.006s
14:22:05.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.144240 26793 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:05.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.146205 26772 fs_manager.cc:730] Time spent opening block manager: real 0.011s	user 0.003s	sys 0.009s
14:22:05.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.147128 26772 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:05.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b575acd823514305a1d3e6f006cb011b"
14:22:05.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:05 on dist-test-slave-45s3"
14:22:05.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.148020 26772 fs_report.cc:389] FS layout report
14:22:05.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:05.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:05.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/data
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:05.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:05.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.264089 26772 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.266559 26772 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:05.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.267520 26772 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:05.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.277380 26772 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.290019 26772 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:22:05.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.290774 26772 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.001s	sys 0.000s
14:22:05.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.291340 26772 ts_tablet_manager.cc:616] Registered 0 tablets
14:22:05.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.291769 26772 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:22:05.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.560878 26772 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.129:33083
14:22:05.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.560935 26906 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.129:33083 every 8 connection(s)
14:22:05.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.564836 26772 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:05.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.573326 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 26772
14:22:05.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.573951 26474 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal/instance
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.598495 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/logs
14:22:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.130:0
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.130
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.130
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:05.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:05.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.629638 26907 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:05.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.630036 26911 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:05.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.630266 26907 heartbeater.cc:461] Registering TS with master...
14:22:05.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.630498 26911 heartbeater.cc:461] Registering TS with master...
14:22:05.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.631819 26907 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.633213 26911 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:05.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.634217 26908 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:05.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.634588 26908 heartbeater.cc:461] Registering TS with master...
14:22:05.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.635551 26908 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:05.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.635761 26674 ts_manager.cc:194] Registered new tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:05.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.639300 26594 ts_manager.cc:194] Registered new tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:05.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.640106 26519 ts_manager.cc:194] Registered new tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:05.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:05.642452 26674 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.129:50737
14:22:05.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:05.980567 26917 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:05.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:05.981086 26917 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:05.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:05.981536 26917 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:06.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:06.016269 26917 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:06.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:06.017050 26917 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.130
14:22:06.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:06.054416 26917 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:06.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:06.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:06.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:06.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:06.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data
14:22:06.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:06.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:06.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:06.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.130:0
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.130
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.130
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/logs
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:06.055967 26917 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:06.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:06.057677 26917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:06.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:06.071156 26923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:06.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:06.646147 26907 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:07.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:07.476225 26922 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 26917
14:22:07.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:07.621491 26922 kernel_stack_watchdog.cc:198] Thread 26917 stuck at /home/jenkins-slave/workspace/build_and_test@2/src/kudu/util/thread.cc:642 for 402ms:
14:22:07.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:22:07.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:22:07.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:22:07.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:22:07.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:06.071779 26924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:07.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:07.622728 26917 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.552s	user 0.609s	sys 0.918s
14:22:07.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:07.623049 26917 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.552s	user 0.609s	sys 0.918s
14:22:07.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:07.624290 26926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:07.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:07.628782 26925 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1558 milliseconds
14:22:07.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.628852 26917 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:22:07.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.630303 26917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:07.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.632709 26917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:07.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.634177 26917 hybrid_clock.cc:648] HybridClock initialized: now 1779114127634087 us; error 108 us; skew 500 ppm
14:22:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.635253 26917 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:07.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.645424 26917 webserver.cc:533] Webserver started at http://127.25.218.130:37055/ using document root <none> and password file <none>
14:22:07.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.646538 26917 fs_manager.cc:362] Metadata directory not provided
14:22:07.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.646801 26917 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:07.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.647508 26917 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:22:07.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.652323 26917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/instance:
14:22:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34a6cb78ecdf4d7a85465851c4c3bb04"
14:22:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:07 on dist-test-slave-45s3"
14:22:07.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.653764 26917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal/instance:
14:22:07.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34a6cb78ecdf4d7a85465851c4c3bb04"
14:22:07.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:07 on dist-test-slave-45s3"
14:22:07.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.663863 26917 fs_manager.cc:696] Time spent creating directory manager: real 0.009s	user 0.011s	sys 0.000s
14:22:07.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.669965 26935 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:07.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.671746 26917 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.001s
14:22:07.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.672168 26917 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:07.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34a6cb78ecdf4d7a85465851c4c3bb04"
14:22:07.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:07 on dist-test-slave-45s3"
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.672513 26917 fs_report.cc:389] FS layout report
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/data
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:07.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.743073 26917 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:07.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.744839 26917 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:07.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.745290 26917 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:07.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.748085 26917 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:07.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.752565 26917 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:22:07.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.752787 26917 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:22:07.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.753005 26917 ts_tablet_manager.cc:616] Registered 0 tablets
14:22:07.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.753142 26917 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:22:07.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.925627 26917 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.130:39771
14:22:07.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.925781 27047 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.130:39771 every 8 connection(s)
14:22:07.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.929617 26917 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:07.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.936709 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 26917
14:22:07.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.937510 26474 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal/instance
14:22:07.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.968636 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:07.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:07.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:07.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data
14:22:07.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/logs
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:07.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:07.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.131:0
14:22:07.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.131
14:22:07.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.131
14:22:07.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:22:07.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:07.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:07.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:07.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:07.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:07.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.990170 27050 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:07.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.990689 27050 heartbeater.cc:461] Registering TS with master...
14:22:07.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.990916 27049 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:07.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.991348 27049 heartbeater.cc:461] Registering TS with master...
14:22:07.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.991874 27050 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:07.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.992684 27049 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:07.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.995994 26519 ts_manager.cc:194] Registered new tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:07.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.996898 26594 ts_manager.cc:194] Registered new tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:08.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:07.999615 27048 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:08.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:08.000027 27048 heartbeater.cc:461] Registering TS with master...
14:22:08.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:08.000993 27048 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:08.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:08.003225 26674 ts_manager.cc:194] Registered new tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:08.005204 26674 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.130:33581
14:22:08.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:08.316715 27058 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:08.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:08.317391 27058 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:08.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:08.318120 27058 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:08.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:08.354903 27058 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:08.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:08.356050 27058 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.131
14:22:08.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:08.394805 27058 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:08.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:08.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:08.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:08.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:08.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data
14:22:08.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:08.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.131:0
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.131
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.131
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/logs
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:08.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:08.396692 27058 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:08.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:08.398885 27058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:08.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:08.415110 27065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:09.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.009232 27048 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:09.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:08.419617 27067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:09.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:08.416136 27064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:09.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:09.707660 27066 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
14:22:09.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.707764 27058 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:22:09.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.713039 27058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:09.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.715973 27058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:09.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.717558 27058 hybrid_clock.cc:648] HybridClock initialized: now 1779114129717494 us; error 86 us; skew 500 ppm
14:22:09.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.718556 27058 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:09.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.725450 27058 webserver.cc:533] Webserver started at http://127.25.218.131:39731/ using document root <none> and password file <none>
14:22:09.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.726588 27058 fs_manager.cc:362] Metadata directory not provided
14:22:09.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.726861 27058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:09.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.727628 27058 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:22:09.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.732824 27058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/instance:
14:22:09.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd010bbd952447cb8aab0a8e79691a65"
14:22:09.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:09 on dist-test-slave-45s3"
14:22:09.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.734531 27058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal/instance:
14:22:09.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd010bbd952447cb8aab0a8e79691a65"
14:22:09.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:09 on dist-test-slave-45s3"
14:22:09.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.743566 27058 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.010s	sys 0.001s
14:22:09.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.750456 27075 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:09.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.751979 27058 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.006s	sys 0.001s
14:22:09.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.752457 27058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:09.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd010bbd952447cb8aab0a8e79691a65"
14:22:09.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:09 on dist-test-slave-45s3"
14:22:09.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.752909 27058 fs_report.cc:389] FS layout report
14:22:09.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:09.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:09.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:09.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/data
14:22:09.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:09.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:09.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:09.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:09.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:09.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:09.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:09.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:09.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:09.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:09.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:09.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.806061 27058 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:09.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.807750 27058 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:09.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.808357 27058 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.811321 27058 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:09.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.816210 27058 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:22:09.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.816572 27058 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:22:09.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.816876 27058 ts_tablet_manager.cc:616] Registered 0 tablets
14:22:09.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.817073 27058 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:22:09.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.996766 27058 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.131:44025
14:22:09.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.997041 27189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.131:44025 every 8 connection(s)
14:22:10.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:09.999657 27058 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:10.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.012411 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 27058
14:22:10.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.013038 26474 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal/instance
14:22:10.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.043795 27190 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:10.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.044451 27190 heartbeater.cc:461] Registering TS with master...
14:22:10.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.045919 27190 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:10.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.049203 26674 ts_manager.cc:194] Registered new tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:10.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.051291 26674 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.131:42193
14:22:10.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.051836 27191 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:10.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.052299 27191 heartbeater.cc:461] Registering TS with master...
14:22:10.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.053221 27191 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:10.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.055662 27192 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:10.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.056056 27192 heartbeater.cc:461] Registering TS with master...
14:22:10.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.056213 26594 ts_manager.cc:194] Registered new tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:10.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.056759 27192 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:10.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.059098 26519 ts_manager.cc:194] Registered new tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:10.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:10.070461 26474 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
14:22:10.083 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:22:10.085 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

14:22:10.101 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.25.218.190"
      port: 42765
    }
    bound_http_address {
      host: "127.25.218.190"
      port: 36517
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.25.218.189"
      port: 41079
    }
    bound_http_address {
      host: "127.25.218.189"
      port: 41613
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.25.218.188"
      port: 34713
    }
    bound_http_address {
      host: "127.25.218.188"
      port: 42581
    }
  }
}

14:22:10.106 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

14:22:10.110 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.25.218.129"
      port: 33083
    }
    bound_http_address {
      host: "127.25.218.129"
      port: 40225
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.25.218.130"
      port: 39771
    }
    bound_http_address {
      host: "127.25.218.130"
      port: 37055
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.25.218.131"
      port: 44025
    }
    bound_http_address {
      host: "127.25.218.131"
      port: 39731
    }
  }
}

14:22:10.111 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
14:22:10.320 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:225)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:246)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
14:22:10.377 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
14:22:10.569 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.25.218.188:34713(127.25.218.188:34713)] connecting to peer
14:22:10.641 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.25.218.189:41079(127.25.218.189:41079)] connecting to peer
14:22:10.643 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.25.218.190:42765(127.25.218.190:42765)] connecting to peer
14:22:10.683 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.25.218.188:34713(127.25.218.188:34713)] Successfully connected to peer
14:22:10.683 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.25.218.189:41079(127.25.218.189:41079)] Successfully connected to peer
14:22:10.683 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.25.218.190:42765(127.25.218.190:42765)] Successfully connected to peer
14:22:10.807 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.25.218.188
14:22:10.808 [DEBUG - kudu-nio-2] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.25.218.190
14:22:10.807 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.25.218.189
14:22:11.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.055363 27190 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:11.111 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x9bdb6ade, L:/127.0.0.1:50714 - R:/127.25.218.189:41079] using SASL/PLAIN
14:22:11.111 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xe336fcca, L:/127.0.0.1:36214 - R:/127.25.218.188:34713] using SASL/PLAIN
14:22:11.111 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x5c73c847, L:/127.0.0.1:33966 - R:/127.25.218.190:42765] using SASL/PLAIN
14:22:11.156 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
14:22:11.157 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=630), Traces: [0ms] refreshing cache from master, [26ms] Sub RPC ConnectToMaster: sending RPC to server master-127.25.218.188:34713, [110ms] Sub RPC ConnectToMaster: sending RPC to server master-127.25.218.189:41079, [112ms] Sub RPC ConnectToMaster: sending RPC to server master-127.25.218.190:42765, [600ms] Sub RPC ConnectToMaster: received response from server master-127.25.218.189:41079: OK, [600ms] Sub RPC ConnectToMaster: received response from server master-127.25.218.190:42765: OK, [600ms] Sub RPC ConnectToMaster: received response from server master-127.25.218.188:34713: OK, deferred=null) after lookup
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.216493 26674 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36214:
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513"
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:22:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
14:22:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:22:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:22:11.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:22:11.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
14:22:11.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
14:22:11.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
14:22:11.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
14:22:11.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
14:22:11.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:22:11.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:22:11.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:11.222435 26674 catalog_manager.cc:7059] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
14:22:11.285 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 4eac364d640b4c9cbcb7a73ecf1a55ac
14:22:11.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.313520 26983 tablet_service.cc:1511] Processing CreateTablet for tablet 747213ca655547928c53ff9dc6b4ca6c (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513 [id=4eac364d640b4c9cbcb7a73ecf1a55ac]), partition=RANGE (key) PARTITION UNBOUNDED
14:22:11.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.315670 26983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 747213ca655547928c53ff9dc6b4ca6c. 1 dirs total, 0 dirs full, 0 dirs failed
14:22:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.351783 26842 tablet_service.cc:1511] Processing CreateTablet for tablet 747213ca655547928c53ff9dc6b4ca6c (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513 [id=4eac364d640b4c9cbcb7a73ecf1a55ac]), partition=RANGE (key) PARTITION UNBOUNDED
14:22:11.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.353960 26842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 747213ca655547928c53ff9dc6b4ca6c. 1 dirs total, 0 dirs full, 0 dirs failed
14:22:11.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.353933 27225 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap starting.
14:22:11.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.355214 27123 tablet_service.cc:1511] Processing CreateTablet for tablet 747213ca655547928c53ff9dc6b4ca6c (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513 [id=4eac364d640b4c9cbcb7a73ecf1a55ac]), partition=RANGE (key) PARTITION UNBOUNDED
14:22:11.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.357357 27123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 747213ca655547928c53ff9dc6b4ca6c. 1 dirs total, 0 dirs full, 0 dirs failed
14:22:11.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.364015 27225 tablet_bootstrap.cc:654] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Neither blocks nor log segments found. Creating new log.
14:22:11.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.366784 27225 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Log is configured to *not* fsync() on all Append() calls
14:22:11.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.380975 27225 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: No bootstrap required, opened a new log
14:22:11.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.381584 27225 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Time spent bootstrapping tablet: real 0.029s	user 0.020s	sys 0.004s
14:22:11.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.383857 27227 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Bootstrap starting.
14:22:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.386682 27228 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Bootstrap starting.
14:22:11.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.395246 27228 tablet_bootstrap.cc:654] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Neither blocks nor log segments found. Creating new log.
14:22:11.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.397758 27227 tablet_bootstrap.cc:654] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Neither blocks nor log segments found. Creating new log.
14:22:11.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.398485 27228 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Log is configured to *not* fsync() on all Append() calls
14:22:11.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.400259 27227 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Log is configured to *not* fsync() on all Append() calls
14:22:11.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.404564 27228 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: No bootstrap required, opened a new log
14:22:11.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.405133 27228 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Time spent bootstrapping tablet: real 0.019s	user 0.005s	sys 0.012s
14:22:11.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.407028 27227 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: No bootstrap required, opened a new log
14:22:11.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.407580 27227 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Time spent bootstrapping tablet: real 0.024s	user 0.004s	sys 0.011s
14:22:11.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.413113 27225 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:11.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.414096 27225 raft_consensus.cc:385] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:22:11.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.414438 27225 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a6cb78ecdf4d7a85465851c4c3bb04, State: Initialized, Role: FOLLOWER
14:22:11.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.415458 27225 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:11.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.429162 27225 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Time spent starting tablet: real 0.047s	user 0.028s	sys 0.013s
14:22:11.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.439066 27228 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:11.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.441247 27228 raft_consensus.cc:385] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:22:11.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.441597 27228 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b575acd823514305a1d3e6f006cb011b, State: Initialized, Role: FOLLOWER
14:22:11.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.441088 27231 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:22:11.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.443061 27228 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:11.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.443769 27231 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:11.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.447625 27231 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:11.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.458544 27227 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:11.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.460045 27227 raft_consensus.cc:385] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:22:11.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.460633 27227 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd010bbd952447cb8aab0a8e79691a65, State: Initialized, Role: FOLLOWER
14:22:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.461925 27227 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:11.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.466105 27228 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Time spent starting tablet: real 0.061s	user 0.043s	sys 0.013s
14:22:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.475093 26862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b575acd823514305a1d3e6f006cb011b" is_pre_election: true
14:22:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.476163 26862 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a6cb78ecdf4d7a85465851c4c3bb04 in term 0.
14:22:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.476622 27227 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Time spent starting tablet: real 0.068s	user 0.029s	sys 0.011s
14:22:11.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.480931 26939 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, b575acd823514305a1d3e6f006cb011b; no voters: 
14:22:11.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:11.482434 27055 tablet.cc:2404] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:22:11.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.484709 27231 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 0 FOLLOWER]: Leader pre-election won for term 1
14:22:11.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.485116 27231 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:22:11.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.485455 27231 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 0 FOLLOWER]: Advancing to term 1
14:22:11.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.490912 27145 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd010bbd952447cb8aab0a8e79691a65" is_pre_election: true
14:22:11.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.492388 27145 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a6cb78ecdf4d7a85465851c4c3bb04 in term 0.
14:22:11.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.493499 27231 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:11.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.495023 27231 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 1 election: Requested vote from peers b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.496052 26862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b575acd823514305a1d3e6f006cb011b"
14:22:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.496270 27145 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd010bbd952447cb8aab0a8e79691a65"
14:22:11.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.496675 26862 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 0 FOLLOWER]: Advancing to term 1
14:22:11.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.496753 27145 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 0 FOLLOWER]: Advancing to term 1
14:22:11.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.503950 27145 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a6cb78ecdf4d7a85465851c4c3bb04 in term 1.
14:22:11.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.503950 26862 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a6cb78ecdf4d7a85465851c4c3bb04 in term 1.
14:22:11.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.505110 26939 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, b575acd823514305a1d3e6f006cb011b; no voters: 
14:22:11.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.505810 27231 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 1 FOLLOWER]: Leader election won for term 1
14:22:11.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.507529 27231 raft_consensus.cc:697] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 1 LEADER]: Becoming Leader. State: Replica: 34a6cb78ecdf4d7a85465851c4c3bb04, State: Running, Role: LEADER
14:22:11.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.508632 27231 consensus_queue.cc:237] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:11.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.519081 26674 catalog_manager.cc:5697] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130). New cstate: current_term: 1 leader_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } health_report { overall_health: UNKNOWN } } }
14:22:11.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:11.526656 27194 tablet.cc:2404] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:22:11.591 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 4eac364d640b4c9cbcb7a73ecf1a55ac
14:22:11.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:11.610406 26912 tablet.cc:2404] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:22:11.613 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:11.620 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299994, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:11.621 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [12ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=null) after lookup
14:22:11.632 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:11.634 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] Successfully connected to peer
14:22:11.700 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0x9319391b, L:/127.0.0.1:49930 - R:/127.25.218.130:39771] using TOKEN/null
14:22:11.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.714874 26862 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 1 FOLLOWER]: Refusing update from remote peer 34a6cb78ecdf4d7a85465851c4c3bb04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:22:11.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.714874 27145 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 1 FOLLOWER]: Refusing update from remote peer 34a6cb78ecdf4d7a85465851c4c3bb04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:22:11.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.716876 27236 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
14:22:11.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.717731 27231 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
14:22:11.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.745558 27239 mvcc.cc:204] Tried to move back new op lower bound from 7287251483486433280 to 7287251482669199360. Current Snapshot: MvccSnapshot[applied={T|T < 7287251483486433280}]
14:22:11.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.789642 27242 mvcc.cc:204] Tried to move back new op lower bound from 7287251483486433280 to 7287251482669199360. Current Snapshot: MvccSnapshot[applied={T|T < 7287251483486433280}]
14:22:11.952 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:22:11.954 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.218.130:39771
14:22:11.956 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:22:11.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:11.957099 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 26917
14:22:12.000 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:12.000 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] handling channelInactive
14:22:12.001 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state READY due to: connection closed
14:22:12.002 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: connection closed
14:22:12.017 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:12.017 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:12.018 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [1ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1255884132(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@478b653b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@513a36c4 -> wakeup thread Time-limited test)) after lookup
14:22:12.019 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:12.023 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.023 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.024 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.053 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:12.054 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:12.054 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=53), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [1ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [24ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [24ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [53ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1255884132(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@478b653b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1626727790 after retry RPC@861416675), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@513a36c4 -> wakeup thread Time-limited test -> (continuation of Deferred@1626727790 after retry RPC@861416675))) after lookup
14:22:12.055 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:12.061 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.061 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.062 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.094 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:12.094 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:12.095 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=94), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [1ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [24ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [24ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [53ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [55ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [61ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [62ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [93ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1255884132(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@478b653b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1626727790 after retry RPC@861416675) -> (continuation of Deferred@1658200880 after retry RPC@966081615), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@513a36c4 -> wakeup thread Time-limited test -> (continuation of Deferred@1626727790 after retry RPC@861416675) -> (continuation of Deferred@1658200880 after retry RPC@966081615))) after lookup
14:22:12.096 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:12.098 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.099 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.099 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.233 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:12.234 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:12.234 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=234), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [1ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [24ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [24ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [53ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [55ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [61ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [62ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [93ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [95ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [99ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [99ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [228ms] refreshing cache from master, [228ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [233ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1255884132(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@478b653b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1626727790 after retry RPC@861416675) -> (continuation of Deferred@1658200880 after retry RPC@966081615) -> (continuation of Deferred@1060829476 after retry RPC@764210814), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@513a36c4 -> wakeup thread Time-limited test -> (continuation of Deferred@1626727790 after retry RPC@861416675) -> (continuation of Deferred@1658200880 after retry RPC@966081615) -> (continuation of Deferred@1060829476 after retry RPC@764210814))) after lookup
14:22:12.235 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:12.237 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.238 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.238 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.574 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:12.575 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:12.576 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=574), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [1ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [24ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [24ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [53ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [55ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [61ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [62ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [93ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [95ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [99ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [99ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [228ms] refreshing cache from master, [228ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [233ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [235ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [238ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [238ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [568ms] refreshing cache from master, [568ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [574ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1255884132(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@478b653b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1626727790 after retry RPC@861416675) -> (continuation of Deferred@1658200880 after retry RPC@966081615) -> (continuation of Deferred@1060829476 after retry RPC@764210814) -> (continuation of Deferred@1066797434 after retry RPC@855780980), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@513a36c4 -> wakeup thread Time-limited test -> (continuation of Deferred@1626727790 after retry RPC@861416675) -> (continuation of Deferred@1658200880 after retry RPC@966081615) -> (continuation of Deferred@1060829476 after retry RPC@764210814) -> (continuation of Deferred@1066797434 after retry RPC@855780980))) after lookup
14:22:12.577 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:12.578 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.579 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:12.579 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:13.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.357391 27256 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 34a6cb78ecdf4d7a85465851c4c3bb04)
14:22:13.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.357817 27256 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:13.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.360821 27256 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:13.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.362720 27257 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 34a6cb78ecdf4d7a85465851c4c3bb04)
14:22:13.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.364406 27257 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:13.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:13.366981 27078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:22:13.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:13.373828 27078 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:22:13.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.376789 27257 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:13.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:13.379324 26796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:22:13.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.382196 26862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "bd010bbd952447cb8aab0a8e79691a65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b575acd823514305a1d3e6f006cb011b" is_pre_election: true
14:22:13.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.383136 26862 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd010bbd952447cb8aab0a8e79691a65 in term 1.
14:22:13.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.384876 27079 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b575acd823514305a1d3e6f006cb011b, bd010bbd952447cb8aab0a8e79691a65; no voters: 34a6cb78ecdf4d7a85465851c4c3bb04
14:22:13.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.385754 27256 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 1 FOLLOWER]: Leader pre-election won for term 2
14:22:13.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.386096 27256 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 34a6cb78ecdf4d7a85465851c4c3bb04)
14:22:13.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:13.386072 26796 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:22:13.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.386340 27256 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 1 FOLLOWER]: Advancing to term 2
14:22:13.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.389683 27145 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "b575acd823514305a1d3e6f006cb011b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bd010bbd952447cb8aab0a8e79691a65" is_pre_election: true
14:22:13.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.392028 27256 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:13.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.393244 27145 raft_consensus.cc:2393] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b575acd823514305a1d3e6f006cb011b in current term 2: Already voted for candidate bd010bbd952447cb8aab0a8e79691a65 in this term.
14:22:13.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.394342 27256 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 2 election: Requested vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:13.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.395617 26862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "bd010bbd952447cb8aab0a8e79691a65" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b575acd823514305a1d3e6f006cb011b"
14:22:13.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.396364 26862 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 1 FOLLOWER]: Advancing to term 2
14:22:13.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.397486 26797 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b575acd823514305a1d3e6f006cb011b; no voters: 34a6cb78ecdf4d7a85465851c4c3bb04, bd010bbd952447cb8aab0a8e79691a65
14:22:13.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:13.399457 27078 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:22:13.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.402318 26862 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd010bbd952447cb8aab0a8e79691a65 in term 2.
14:22:13.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.402817 27257 raft_consensus.cc:2749] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
14:22:13.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.403581 27079 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b575acd823514305a1d3e6f006cb011b, bd010bbd952447cb8aab0a8e79691a65; no voters: 34a6cb78ecdf4d7a85465851c4c3bb04
14:22:13.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.404279 27256 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 2 FOLLOWER]: Leader election won for term 2
14:22:13.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.406296 27256 raft_consensus.cc:697] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 2 LEADER]: Becoming Leader. State: Replica: bd010bbd952447cb8aab0a8e79691a65, State: Running, Role: LEADER
14:22:13.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.407544 27256 consensus_queue.cc:237] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:13.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.416662 26673 catalog_manager.cc:5697] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 reported cstate change: term changed from 1 to 2, leader changed from 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130) to bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131). New cstate: current_term: 2 leader_uuid: "bd010bbd952447cb8aab0a8e79691a65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } health_report { overall_health: HEALTHY } } }
14:22:13.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.820680 26862 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 2 FOLLOWER]: Refusing update from remote peer bd010bbd952447cb8aab0a8e79691a65: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
14:22:13.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:13.822078 27256 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
14:22:13.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:13.838855 27078 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 -> Peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Couldn't send request to peer 34a6cb78ecdf4d7a85465851c4c3bb04. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:22:14.293 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:14.294 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:14.294 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2294), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [1ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [24ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [24ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [53ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [55ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [61ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [62ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [93ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [95ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [99ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [99ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [228ms] refreshing cache from master, [228ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [233ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [235ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [238ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [238ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [568ms] refreshing cache from master, [568ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [574ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [577ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [579ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [579ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [2288ms] refreshing cache from master, [2288ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [2293ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1255884132(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@478b653b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1626727790 after retry RPC@861416675) -> (continuation of Deferred@1658200880 after retry RPC@966081615) -> (continuation of Deferred@1060829476 after retry RPC@764210814) -> (continuation of Deferred@1066797434 after retry RPC@855780980) -> (continuation of Deferred@937755788 after retry RPC@570204409), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@513a36c4 -> wakeup thread Time-limited test -> (continuation of Deferred@1626727790 after retry RPC@861416675) -> (continuation of Deferred@1658200880 after retry RPC@966081615) -> (continuation of Deferred@1060829476 after retry RPC@764210814) -> (continuation of Deferred@1066797434 after retry RPC@855780980) -> (continuation of Deferred@937755788 after retry RPC@570204409))) after lookup
14:22:14.295 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:14.297 [DEBUG - kudu-nio-1] (Connection.java:757) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] Successfully connected to peer
14:22:14.322 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x07280b49, L:/127.0.0.1:40568 - R:/127.25.218.131:44025] using TOKEN/null
14:22:14.374 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.218.130:39771
14:22:14.376 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:14.377646 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/logs
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.130:39771
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.130
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37055
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.130
14:22:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:14.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:14.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:14.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:14.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:14.752630 27285 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:14.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:14.753139 27285 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:14.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:14.753580 27285 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:14.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:14.797183 27285 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:14.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:14.798130 27285 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.130
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:14.839350 27285 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.130:39771
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.130
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37055
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.130
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/logs
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:14.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:14.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:14.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:14.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:14.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:14.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:14.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:14.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:14.841274 27285 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:14.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:14.843137 27285 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:14.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:14.862984 27291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:16.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:16.211194 27078 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 -> Peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Couldn't send request to peer 34a6cb78ecdf4d7a85465851c4c3bb04. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:22:16.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:16.266144 27290 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 27285
14:22:16.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:16.837067 27290 kernel_stack_watchdog.cc:198] Thread 27285 stuck at /home/jenkins-slave/workspace/build_and_test@2/src/kudu/util/thread.cc:642 for 400ms:
14:22:16.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:22:16.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:22:16.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:22:16.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:22:16.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:14.863155 27292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:16.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:16.848958 27293 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1986 milliseconds
14:22:16.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:16.848981 27285 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.987s	user 0.000s	sys 0.002s
14:22:16.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:16.849435 27285 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.988s	user 0.000s	sys 0.003s
14:22:16.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:16.851490 27295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:16.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.851613 27285 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:22:16.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.853008 27285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:16.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.855792 27285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:16.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.857335 27285 hybrid_clock.cc:648] HybridClock initialized: now 1779114136857285 us; error 43 us; skew 500 ppm
14:22:16.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.858397 27285 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:16.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.866950 27285 webserver.cc:533] Webserver started at http://127.25.218.130:37055/ using document root <none> and password file <none>
14:22:16.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.868469 27285 fs_manager.cc:362] Metadata directory not provided
14:22:16.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.870129 27285 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:16.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.891080 27285 fs_manager.cc:714] Time spent opening directory manager: real 0.012s	user 0.005s	sys 0.003s
14:22:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.898551 27309 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:16.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.900853 27285 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.004s	sys 0.002s
14:22:16.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.901337 27285 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:16.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34a6cb78ecdf4d7a85465851c4c3bb04"
14:22:16.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:07 on dist-test-slave-45s3"
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.903482 27285 fs_report.cc:389] FS layout report
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/data
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:16.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.967865 27285 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:16.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.969789 27285 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:16.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.970366 27285 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:16.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.973670 27285 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:16.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.980094 27316 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:22:16.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.989046 27285 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:22:16.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.989388 27285 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
14:22:16.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.989761 27285 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:22:16.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.996373 27285 ts_tablet_manager.cc:616] Registered 1 tablets
14:22:16.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.996623 27285 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.002s	sys 0.005s
14:22:16.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:16.997102 27316 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap starting.
14:22:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.063788 27316 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Log is configured to *not* fsync() on all Append() calls
14:22:17.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.092859 27316 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:22:17.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.093885 27316 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap complete.
14:22:17.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.095883 27316 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Time spent bootstrapping tablet: real 0.099s	user 0.057s	sys 0.040s
14:22:17.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.118247 27316 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:17.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.122201 27316 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a6cb78ecdf4d7a85465851c4c3bb04, State: Initialized, Role: FOLLOWER
14:22:17.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.123430 27316 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:17.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:17.128446 26550 debug-util.cc:398] Leaking SignalData structure 0x7b08000703c0 after lost signal to thread 26487
14:22:17.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.128739 27316 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Time spent starting tablet: real 0.032s	user 0.027s	sys 0.005s
14:22:17.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:17.129326 26550 debug-util.cc:398] Leaking SignalData structure 0x7b080009c5c0 after lost signal to thread 26553
14:22:17.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.203379 27285 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.130:39771
14:22:17.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.203799 27425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.130:39771 every 8 connection(s)
14:22:17.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.207629 27285 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:17.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.217196 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 27285
14:22:17.218 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:17.239 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:22:17.240 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.218.131:44025
14:22:17.241 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:22:17.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.241771 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 27058
14:22:17.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.257042 27427 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:17.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.257663 27427 heartbeater.cc:461] Registering TS with master...
14:22:17.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.259181 27427 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:17.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.263341 27426 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:17.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.263800 27426 heartbeater.cc:461] Registering TS with master...
14:22:17.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.264909 27426 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:17.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.269193 26593 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:17.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.274523 26673 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:17.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.279071 26673 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.130:49563
14:22:17.285 [DEBUG - kudu-nio-1] (Connection.java:239) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] handling channelInactive
14:22:17.285 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state READY due to: connection closed
14:22:17.298 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:17.300 [DEBUG - Time-limited test] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:17.301 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.302 [DEBUG - kudu-nio-2] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.302 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.305104 27428 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:17.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.305668 27428 heartbeater.cc:461] Registering TS with master...
14:22:17.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.306633 27428 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:17.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:17.312839 26519 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:17.316 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:17.317 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:17.318 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@54405217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@4701ab97 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1720bdd4 -> wakeup thread Time-limited test)) after lookup
14:22:17.320 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:17.322 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.322 [DEBUG - kudu-nio-3] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.322 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.354 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:17.355 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:17.355 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=56), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [23ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [54ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@54405217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@4701ab97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@925897751 after retry RPC@636272952), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1720bdd4 -> wakeup thread Time-limited test -> (continuation of Deferred@925897751 after retry RPC@636272952))) after lookup
14:22:17.356 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:17.358 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.358 [DEBUG - kudu-nio-4] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.359 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.394 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:17.394 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:17.395 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=96), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [23ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [54ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [57ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [59ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [60ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [94ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@54405217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@4701ab97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@925897751 after retry RPC@636272952) -> (continuation of Deferred@438920991 after retry RPC@1502580835), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1720bdd4 -> wakeup thread Time-limited test -> (continuation of Deferred@925897751 after retry RPC@636272952) -> (continuation of Deferred@438920991 after retry RPC@1502580835))) after lookup
14:22:17.396 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:17.398 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.398 [DEBUG - kudu-nio-5] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.398 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.433 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:17.433 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:17.434 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=134), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [23ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [54ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [57ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [59ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [60ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [94ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [97ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [99ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [99ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [129ms] refreshing cache from master, [129ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [133ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@54405217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@4701ab97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@925897751 after retry RPC@636272952) -> (continuation of Deferred@438920991 after retry RPC@1502580835) -> (continuation of Deferred@83807706 after retry RPC@559368622), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1720bdd4 -> wakeup thread Time-limited test -> (continuation of Deferred@925897751 after retry RPC@636272952) -> (continuation of Deferred@438920991 after retry RPC@1502580835) -> (continuation of Deferred@83807706 after retry RPC@559368622))) after lookup
14:22:17.434 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:17.436 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.436 [DEBUG - kudu-nio-6] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.436 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.753 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:17.754 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:17.755 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=455), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [23ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [54ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [57ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [59ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [60ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [94ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [97ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [99ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [99ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [129ms] refreshing cache from master, [129ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [133ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [135ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [137ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [137ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [449ms] refreshing cache from master, [450ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [454ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@54405217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@4701ab97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@925897751 after retry RPC@636272952) -> (continuation of Deferred@438920991 after retry RPC@1502580835) -> (continuation of Deferred@83807706 after retry RPC@559368622) -> (continuation of Deferred@926976 after retry RPC@453969558), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1720bdd4 -> wakeup thread Time-limited test -> (continuation of Deferred@925897751 after retry RPC@636272952) -> (continuation of Deferred@438920991 after retry RPC@1502580835) -> (continuation of Deferred@83807706 after retry RPC@559368622) -> (continuation of Deferred@926976 after retry RPC@453969558))) after lookup
14:22:17.756 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:17.758 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.758 [DEBUG - kudu-nio-7] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:17.758 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:18.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.284823 27426 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.304347 27438 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bd010bbd952447cb8aab0a8e79691a65)
14:22:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.304847 27438 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:18.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.310663 27438 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:18.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:18.315658 26797 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025): Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111)
14:22:18.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.325896 27380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "b575acd823514305a1d3e6f006cb011b" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" is_pre_election: true
14:22:18.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.327061 27380 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b575acd823514305a1d3e6f006cb011b in term 1.
14:22:18.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.328876 26796 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, b575acd823514305a1d3e6f006cb011b; no voters: bd010bbd952447cb8aab0a8e79691a65
14:22:18.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.329967 27438 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 2 FOLLOWER]: Leader pre-election won for term 3
14:22:18.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.330369 27438 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 2 FOLLOWER]: Starting leader election (detected failure of leader bd010bbd952447cb8aab0a8e79691a65)
14:22:18.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.330757 27438 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 2 FOLLOWER]: Advancing to term 3
14:22:18.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.335891 27438 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:18.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.337960 27438 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 3 election: Requested vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:18.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.338922 27380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "b575acd823514305a1d3e6f006cb011b" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04"
14:22:18.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.339550 27380 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 1 FOLLOWER]: Advancing to term 3
14:22:18.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:18.342634 26797 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025): Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111)
14:22:18.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.347288 27380 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b575acd823514305a1d3e6f006cb011b in term 3.
14:22:18.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.348526 26796 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, b575acd823514305a1d3e6f006cb011b; no voters: bd010bbd952447cb8aab0a8e79691a65
14:22:18.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.349704 27438 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 3 FOLLOWER]: Leader election won for term 3
14:22:18.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.351015 27438 raft_consensus.cc:697] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 3 LEADER]: Becoming Leader. State: Replica: b575acd823514305a1d3e6f006cb011b, State: Running, Role: LEADER
14:22:18.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.351950 27438 consensus_queue.cc:237] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:18.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.361310 26673 catalog_manager.cc:5697] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b reported cstate change: term changed from 2 to 3, leader changed from bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131) to b575acd823514305a1d3e6f006cb011b (127.25.218.129). New cstate: current_term: 3 leader_uuid: "b575acd823514305a1d3e6f006cb011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } health_report { overall_health: UNKNOWN } } }
14:22:18.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.733695 27380 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 3 FOLLOWER]: Refusing update from remote peer b575acd823514305a1d3e6f006cb011b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
14:22:18.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:18.735783 27438 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.001s
14:22:18.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:18.780441 26797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111) [suppressed 2 similar messages]
14:22:18.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:18.788705 26797 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b -> Peer bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025): Couldn't send request to peer bd010bbd952447cb8aab0a8e79691a65. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:22:19.094 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:19.095 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:19.096 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1796), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [23ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [54ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [57ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [59ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [60ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [94ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [97ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [99ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [99ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [129ms] refreshing cache from master, [129ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [133ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [135ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [137ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [137ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [449ms] refreshing cache from master, [450ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [454ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [456ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [459ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [459ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [1789ms] refreshing cache from master, [1790ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [1795ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@54405217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@4701ab97 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@925897751 after retry RPC@636272952) -> (continuation of Deferred@438920991 after retry RPC@1502580835) -> (continuation of Deferred@83807706 after retry RPC@559368622) -> (continuation of Deferred@926976 after retry RPC@453969558) -> (continuation of Deferred@540781891 after retry RPC@1927232213), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1720bdd4 -> wakeup thread Time-limited test -> (continuation of Deferred@925897751 after retry RPC@636272952) -> (continuation of Deferred@438920991 after retry RPC@1502580835) -> (continuation of Deferred@83807706 after retry RPC@559368622) -> (continuation of Deferred@926976 after retry RPC@453969558) -> (continuation of Deferred@540781891 after retry RPC@1927232213))) after lookup
14:22:19.097 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:19.098 [DEBUG - kudu-nio-0] (Connection.java:757) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] Successfully connected to peer
14:22:19.122 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x2a51b0c9, L:/127.0.0.1:49538 - R:/127.25.218.129:33083] using TOKEN/null
14:22:19.178 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.218.131:44025
14:22:19.180 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:22:19.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:19.181356 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:19.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/logs
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.131:44025
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.131
14:22:19.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:19.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39731
14:22:19.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.131
14:22:19.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:19.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:19.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:19.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:19.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:19.540134 27465 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:19.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:19.540623 27465 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:19.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:19.541251 27465 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:19.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:19.576383 27465 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:19.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:19.577477 27465 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.131
14:22:19.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:19.619973 27465 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:19.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:19.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:19.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:19.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:19.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data
14:22:19.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:19.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.131:44025
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.131
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39731
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.131
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/logs
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:19.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:19.622080 27465 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:19.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:19.624521 27465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:19.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:19.640275 27474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:20.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:19.641836 27475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:20.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:20.993974 27477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:20.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:20.996426 27476 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1350 milliseconds
14:22:20.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:20.996680 27465 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:22:20.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:20.998174 27465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:21.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.001109 27465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:21.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.002765 27465 hybrid_clock.cc:648] HybridClock initialized: now 1779114141002633 us; error 147 us; skew 500 ppm
14:22:21.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.003844 27465 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:21.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.012326 27465 webserver.cc:533] Webserver started at http://127.25.218.131:39731/ using document root <none> and password file <none>
14:22:21.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.013561 27465 fs_manager.cc:362] Metadata directory not provided
14:22:21.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.013891 27465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:21.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.022869 27465 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
14:22:21.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.028928 27486 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:21.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.030403 27465 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.000s
14:22:21.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.030797 27465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:21.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd010bbd952447cb8aab0a8e79691a65"
14:22:21.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:09 on dist-test-slave-45s3"
14:22:21.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.033260 27465 fs_report.cc:389] FS layout report
14:22:21.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:21.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:21.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/data
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:21.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.105329 27465 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:21.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.106920 27465 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:21.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.107383 27465 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.110288 27465 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:21.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.116532 27493 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:22:21.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.125349 27465 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:22:21.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.125598 27465 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
14:22:21.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.125882 27465 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:22:21.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.131733 27465 ts_tablet_manager.cc:616] Registered 1 tablets
14:22:21.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.131997 27465 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.006s	sys 0.001s
14:22:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.132596 27493 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Bootstrap starting.
14:22:21.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:21.182607 26797 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b -> Peer bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025): Couldn't send request to peer bd010bbd952447cb8aab0a8e79691a65. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:22:21.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.204767 27493 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Log is configured to *not* fsync() on all Append() calls
14:22:21.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.264837 27493 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:22:21.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.266095 27493 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Bootstrap complete.
14:22:21.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.267822 27493 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Time spent bootstrapping tablet: real 0.136s	user 0.079s	sys 0.050s
14:22:21.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.291309 27493 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:21.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.295408 27493 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd010bbd952447cb8aab0a8e79691a65, State: Initialized, Role: FOLLOWER
14:22:21.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.296887 27493 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:21.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.303385 27493 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Time spent starting tablet: real 0.035s	user 0.024s	sys 0.008s
14:22:21.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.347187 27465 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.131:44025
14:22:21.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.347488 27601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.131:44025 every 8 connection(s)
14:22:21.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.350616 27465 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:21.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.361449 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 27465
14:22:21.363 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:21.389 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:22:21.390 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.218.129:33083
14:22:21.391 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:22:21.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.391566 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 26772
14:22:21.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.416311 27602 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:21.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.416311 27604 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:21.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.416337 27603 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:21.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.416992 27602 heartbeater.cc:461] Registering TS with master...
14:22:21.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.417052 27603 heartbeater.cc:461] Registering TS with master...
14:22:21.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.417049 27604 heartbeater.cc:461] Registering TS with master...
14:22:21.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.418640 27603 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:21.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.419068 27602 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:21.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.418640 27604 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:21.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.424662 26593 ts_manager.cc:194] Re-registered known tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:21.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.425083 26518 ts_manager.cc:194] Re-registered known tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:21.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.425642 26674 ts_manager.cc:194] Re-registered known tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:21.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:21.430346 26674 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.131:44303
14:22:21.435 [DEBUG - kudu-nio-0] (Connection.java:239) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] handling channelInactive
14:22:21.435 [DEBUG - kudu-nio-0] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state READY due to: connection closed
14:22:21.436 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:21.437 [DEBUG - Time-limited test] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:21.439 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.439 [DEBUG - kudu-nio-1] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.440 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.453 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:21.454 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:21.454 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [2ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1019276763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@463be033 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@232f3c7e -> wakeup thread Time-limited test)) after lookup
14:22:21.455 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:21.457 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.457 [DEBUG - kudu-nio-2] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.457 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.475 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:21.475 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:21.475 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [2ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [20ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [20ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [37ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1019276763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@463be033 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@656699722 after retry RPC@2108847388), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@232f3c7e -> wakeup thread Time-limited test -> (continuation of Deferred@656699722 after retry RPC@2108847388))) after lookup
14:22:21.477 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:21.478 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.478 [DEBUG - kudu-nio-3] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.478 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.513 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:21.513 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:21.514 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=77), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [2ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [20ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [20ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [37ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [39ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [41ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [41ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [76ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1019276763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@463be033 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@656699722 after retry RPC@2108847388) -> (continuation of Deferred@1858097410 after retry RPC@3461971), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@232f3c7e -> wakeup thread Time-limited test -> (continuation of Deferred@656699722 after retry RPC@2108847388) -> (continuation of Deferred@1858097410 after retry RPC@3461971))) after lookup
14:22:21.514 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:21.515 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.516 [DEBUG - kudu-nio-4] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.516 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.574 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:21.574 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:21.574 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=137), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [2ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [20ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [20ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [37ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [39ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [41ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [41ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [76ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [77ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [79ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [79ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [131ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [136ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1019276763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@463be033 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@656699722 after retry RPC@2108847388) -> (continuation of Deferred@1858097410 after retry RPC@3461971) -> (continuation of Deferred@1527242219 after retry RPC@2106536851), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@232f3c7e -> wakeup thread Time-limited test -> (continuation of Deferred@656699722 after retry RPC@2108847388) -> (continuation of Deferred@1858097410 after retry RPC@3461971) -> (continuation of Deferred@1527242219 after retry RPC@2106536851))) after lookup
14:22:21.575 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:21.577 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.577 [DEBUG - kudu-nio-5] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.577 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.714 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:21.714 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:21.715 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=277), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [2ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [20ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [20ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [37ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [39ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [41ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [41ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [76ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [77ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [79ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [79ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [131ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [136ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [138ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [140ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [140ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [271ms] refreshing cache from master, [271ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [276ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1019276763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@463be033 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@656699722 after retry RPC@2108847388) -> (continuation of Deferred@1858097410 after retry RPC@3461971) -> (continuation of Deferred@1527242219 after retry RPC@2106536851) -> (continuation of Deferred@1552205700 after retry RPC@1486058540), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@232f3c7e -> wakeup thread Time-limited test -> (continuation of Deferred@656699722 after retry RPC@2108847388) -> (continuation of Deferred@1858097410 after retry RPC@3461971) -> (continuation of Deferred@1527242219 after retry RPC@2106536851) -> (continuation of Deferred@1552205700 after retry RPC@1486058540))) after lookup
14:22:21.715 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:21.717 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.717 [DEBUG - kudu-nio-6] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:21.717 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:22.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.434816 27602 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:22.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.883656 27614 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b575acd823514305a1d3e6f006cb011b)
14:22:22.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.884330 27614 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:22.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.887159 27614 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:22.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:22.890669 27313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111)
14:22:22.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:22.896493 27313 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111)
14:22:22.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.903086 27556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "bd010bbd952447cb8aab0a8e79691a65" is_pre_election: true
14:22:22.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.903966 27556 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a6cb78ecdf4d7a85465851c4c3bb04 in term 2.
14:22:22.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.905719 27313 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, bd010bbd952447cb8aab0a8e79691a65; no voters: b575acd823514305a1d3e6f006cb011b
14:22:22.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.906435 27614 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 3 FOLLOWER]: Leader pre-election won for term 4
14:22:22.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.906777 27614 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b575acd823514305a1d3e6f006cb011b)
14:22:22.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.907007 27614 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 3 FOLLOWER]: Advancing to term 4
14:22:22.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.912314 27614 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.914194 27614 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 4 election: Requested vote from peers b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:22.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.916342 27556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "bd010bbd952447cb8aab0a8e79691a65"
14:22:22.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.916942 27556 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 2 FOLLOWER]: Advancing to term 4
14:22:22.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:22.919646 27313 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111)
14:22:22.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.925375 27556 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a6cb78ecdf4d7a85465851c4c3bb04 in term 4.
14:22:22.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.926689 27313 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, bd010bbd952447cb8aab0a8e79691a65; no voters: b575acd823514305a1d3e6f006cb011b
14:22:22.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.927562 27614 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 4 FOLLOWER]: Leader election won for term 4
14:22:22.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.929585 27614 raft_consensus.cc:697] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 4 LEADER]: Becoming Leader. State: Replica: 34a6cb78ecdf4d7a85465851c4c3bb04, State: Running, Role: LEADER
14:22:22.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.930617 27614 consensus_queue.cc:237] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:22.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:22.943085 26674 catalog_manager.cc:5697] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 reported cstate change: term changed from 3 to 4, leader changed from b575acd823514305a1d3e6f006cb011b (127.25.218.129) to 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130). New cstate: current_term: 4 leader_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } health_report { overall_health: UNKNOWN } } }
14:22:23.113 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:23.114 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:23.114 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1677), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [2ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [16ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [20ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [20ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [37ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [39ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [41ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [41ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [76ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [77ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [79ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [79ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [131ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [136ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [138ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [140ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [140ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [271ms] refreshing cache from master, [271ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [276ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [278ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [280ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [280ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [1671ms] refreshing cache from master, [1672ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [1676ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1019276763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@463be033 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@656699722 after retry RPC@2108847388) -> (continuation of Deferred@1858097410 after retry RPC@3461971) -> (continuation of Deferred@1527242219 after retry RPC@2106536851) -> (continuation of Deferred@1552205700 after retry RPC@1486058540) -> (continuation of Deferred@1261426770 after retry RPC@148825871), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@232f3c7e -> wakeup thread Time-limited test -> (continuation of Deferred@656699722 after retry RPC@2108847388) -> (continuation of Deferred@1858097410 after retry RPC@3461971) -> (continuation of Deferred@1527242219 after retry RPC@2106536851) -> (continuation of Deferred@1552205700 after retry RPC@1486058540) -> (continuation of Deferred@1261426770 after retry RPC@148825871))) after lookup
14:22:23.115 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:23.116 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] Successfully connected to peer
14:22:23.150 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0x6d234fdc, L:/127.0.0.1:45570 - R:/127.25.218.130:39771] using TOKEN/null
14:22:23.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:23.174912 27556 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 4 FOLLOWER]: Refusing update from remote peer 34a6cb78ecdf4d7a85465851c4c3bb04: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
14:22:23.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:23.177337 27614 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
14:22:23.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:23.189144 27313 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 -> Peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Couldn't send request to peer b575acd823514305a1d3e6f006cb011b. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:22:23.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:23.206936 27636 mvcc.cc:204] Tried to move back new op lower bound from 7287251513871839232 to 7287251510699274240. Current Snapshot: MvccSnapshot[applied={T|T < 7287251513871839232}]
14:22:23.267 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.218.129:33083
14:22:23.267 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:22:23.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:23.269019 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:23.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:23.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:23.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/logs
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.129:33083
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.129
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40225
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.129
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:23.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:23.480144 27421 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8f80 after lost signal to thread 27286
14:22:23.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:23.649837 27642 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:23.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:23.650513 27642 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:23.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:23.651283 27642 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:23.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:23.695489 27642 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:23.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:23.696668 27642 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.129
14:22:23.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:23.738821 27642 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:23.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:23.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:23.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:23.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:23.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.129:33083
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.129
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40225
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:23.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.129
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/logs
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:23.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:23.740939 27642 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:23.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:23.742835 27642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:23.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:23.759445 27651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:25.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:23.765506 27652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:25.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:23.767292 27642 server_base.cc:1061] running on GCE node
14:22:25.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:23.765156 27654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:25.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.182225 27642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:25.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.186015 27642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:25.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.187804 27642 hybrid_clock.cc:648] HybridClock initialized: now 1779114145187659 us; error 54 us; skew 500 ppm
14:22:25.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.188819 27642 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:25.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.197145 27642 webserver.cc:533] Webserver started at http://127.25.218.129:40225/ using document root <none> and password file <none>
14:22:25.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.198315 27642 fs_manager.cc:362] Metadata directory not provided
14:22:25.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.198650 27642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:25.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.209074 27642 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.007s	sys 0.001s
14:22:25.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.217176 27667 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:25.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.219046 27642 fs_manager.cc:730] Time spent opening block manager: real 0.007s	user 0.004s	sys 0.000s
14:22:25.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.219496 27642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:25.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b575acd823514305a1d3e6f006cb011b"
14:22:25.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:05 on dist-test-slave-45s3"
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.222009 27642 fs_report.cc:389] FS layout report
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/data
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:25.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:25.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.294876 27642 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:25.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.296846 27642 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:25.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.297441 27642 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:25.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.301352 27642 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:25.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.309530 27674 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:22:25.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.325901 27642 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:22:25.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.326303 27642 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.019s	user 0.003s	sys 0.000s
14:22:25.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.327270 27642 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:22:25.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.337230 27674 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Bootstrap starting.
14:22:25.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.337248 27642 ts_tablet_manager.cc:616] Registered 1 tablets
14:22:25.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.337895 27642 ts_tablet_manager.cc:595] Time spent register tablets: real 0.011s	user 0.009s	sys 0.000s
14:22:25.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.421245 27674 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Log is configured to *not* fsync() on all Append() calls
14:22:25.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.511253 27674 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:22:25.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.512789 27674 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Bootstrap complete.
14:22:25.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.515533 27674 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Time spent bootstrapping tablet: real 0.179s	user 0.105s	sys 0.060s
14:22:25.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.551115 27674 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:25.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.555516 27674 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: b575acd823514305a1d3e6f006cb011b, State: Initialized, Role: FOLLOWER
14:22:25.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.557240 27674 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:25.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.564378 27674 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Time spent starting tablet: real 0.048s	user 0.042s	sys 0.005s
14:22:25.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:25.686358 27313 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 -> Peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Couldn't send request to peer b575acd823514305a1d3e6f006cb011b. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:22:25.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.693024 27642 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.129:33083
14:22:25.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.693949 27783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.129:33083 every 8 connection(s)
14:22:25.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.697374 27642 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:25.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.699908 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 27642
14:22:25.702 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:25.760 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:22:25.771 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.218.130:39771
14:22:25.772 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:22:25.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.772915 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 27285
14:22:25.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.789381 27784 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:25.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.789932 27784 heartbeater.cc:461] Registering TS with master...
14:22:25.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.791410 27784 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:25.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.796885 26674 ts_manager.cc:194] Re-registered known tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:25.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.797482 27785 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:25.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.797845 27785 heartbeater.cc:461] Registering TS with master...
14:22:25.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.798568 27789 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:25.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.799125 27789 heartbeater.cc:461] Registering TS with master...
14:22:25.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.799357 27785 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:25.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.800350 27789 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:25.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.803746 26593 ts_manager.cc:194] Re-registered known tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:25.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.805182 26518 ts_manager.cc:194] Re-registered known tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:25.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:25.805987 26674 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.129:46181
14:22:25.822 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:25.822 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] handling channelInactive
14:22:25.822 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state READY due to: connection closed
14:22:25.830 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:25.832 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.832 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.832 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.852 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:25.853 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:25.853 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=3, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [10ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [10ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1513515716(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@330b7178 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad6eeee -> wakeup thread Time-limited test)) after lookup
14:22:25.853 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:25.855 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.855 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.855 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.873 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:25.874 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:25.874 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=5, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [10ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [10ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [31ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [33ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [33ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [51ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1513515716(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@330b7178 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@115228619 after retry RPC@459407456), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad6eeee -> wakeup thread Time-limited test -> (continuation of Deferred@115228619 after retry RPC@459407456))) after lookup
14:22:25.875 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:25.876 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.877 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.877 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.913 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:25.913 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:25.914 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=7, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=91), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [10ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [10ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [31ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [33ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [33ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [51ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [53ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [55ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [55ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [87ms] refreshing cache from master, [87ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [91ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1513515716(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@330b7178 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@115228619 after retry RPC@459407456) -> (continuation of Deferred@1280272357 after retry RPC@1193615116), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad6eeee -> wakeup thread Time-limited test -> (continuation of Deferred@115228619 after retry RPC@459407456) -> (continuation of Deferred@1280272357 after retry RPC@1193615116))) after lookup
14:22:25.914 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:25.915 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.916 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.916 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.993 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:25.993 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:25.994 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=9, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=171), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [10ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [10ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [31ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [33ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [33ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [51ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [53ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [55ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [55ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [87ms] refreshing cache from master, [87ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [91ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [92ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [94ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [94ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [166ms] refreshing cache from master, [166ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [170ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1513515716(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@330b7178 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@115228619 after retry RPC@459407456) -> (continuation of Deferred@1280272357 after retry RPC@1193615116) -> (continuation of Deferred@448833684 after retry RPC@1978800575), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad6eeee -> wakeup thread Time-limited test -> (continuation of Deferred@115228619 after retry RPC@459407456) -> (continuation of Deferred@1280272357 after retry RPC@1193615116) -> (continuation of Deferred@448833684 after retry RPC@1978800575))) after lookup
14:22:25.995 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:25.996 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.996 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:25.996 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:26.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:26.399653 26705 debug-util.cc:398] Leaking SignalData structure 0x7b08000897c0 after lost signal to thread 26636
14:22:26.473 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:26.473 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:26.474 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=11, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=651), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [10ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [10ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [31ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [33ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [33ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [51ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [53ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [55ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [55ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [87ms] refreshing cache from master, [87ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [91ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [92ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [94ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [94ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [166ms] refreshing cache from master, [166ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [170ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [172ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [174ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [174ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [646ms] refreshing cache from master, [647ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [650ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1513515716(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@330b7178 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@115228619 after retry RPC@459407456) -> (continuation of Deferred@1280272357 after retry RPC@1193615116) -> (continuation of Deferred@448833684 after retry RPC@1978800575) -> (continuation of Deferred@1257496567 after retry RPC@791222519), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad6eeee -> wakeup thread Time-limited test -> (continuation of Deferred@115228619 after retry RPC@459407456) -> (continuation of Deferred@1280272357 after retry RPC@1193615116) -> (continuation of Deferred@448833684 after retry RPC@1978800575) -> (continuation of Deferred@1257496567 after retry RPC@791222519))) after lookup
14:22:26.474 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:26.476 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:26.476 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:26.476 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:26.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.731617 27795 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 34a6cb78ecdf4d7a85465851c4c3bb04)
14:22:26.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.732259 27795 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:26.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.735608 27795 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:26.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:26.743381 27489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:22:26.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:26.750308 27489 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:22:26.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.753389 27739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "bd010bbd952447cb8aab0a8e79691a65" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "b575acd823514305a1d3e6f006cb011b" is_pre_election: true
14:22:26.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.754261 27739 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd010bbd952447cb8aab0a8e79691a65 in term 3.
14:22:26.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.756318 27490 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b575acd823514305a1d3e6f006cb011b, bd010bbd952447cb8aab0a8e79691a65; no voters: 34a6cb78ecdf4d7a85465851c4c3bb04
14:22:26.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.758105 27795 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 4 FOLLOWER]: Leader pre-election won for term 5
14:22:26.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.758524 27795 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 34a6cb78ecdf4d7a85465851c4c3bb04)
14:22:26.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.758893 27795 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 4 FOLLOWER]: Advancing to term 5
14:22:26.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.768199 27795 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.770522 27795 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 5 election: Requested vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.771847 27739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "bd010bbd952447cb8aab0a8e79691a65" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "b575acd823514305a1d3e6f006cb011b"
14:22:26.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.773032 27739 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 3 FOLLOWER]: Advancing to term 5
14:22:26.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:26.775420 27489 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:22:26.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.785391 27739 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd010bbd952447cb8aab0a8e79691a65 in term 5.
14:22:26.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.786907 27490 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b575acd823514305a1d3e6f006cb011b, bd010bbd952447cb8aab0a8e79691a65; no voters: 34a6cb78ecdf4d7a85465851c4c3bb04
14:22:26.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.787884 27795 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 5 FOLLOWER]: Leader election won for term 5
14:22:26.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.790091 27795 raft_consensus.cc:697] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 5 LEADER]: Becoming Leader. State: Replica: bd010bbd952447cb8aab0a8e79691a65, State: Running, Role: LEADER
14:22:26.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.791162 27795 consensus_queue.cc:237] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:26.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.803638 26674 catalog_manager.cc:5697] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 reported cstate change: term changed from 4 to 5, leader changed from 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130) to bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131). New cstate: current_term: 5 leader_uuid: "bd010bbd952447cb8aab0a8e79691a65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } health_report { overall_health: HEALTHY } } }
14:22:26.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:26.810467 27784 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:27.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:27.368316 27489 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 -> Peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Couldn't send request to peer 34a6cb78ecdf4d7a85465851c4c3bb04. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:22:27.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:27.391286 27739 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 5 FOLLOWER]: Refusing update from remote peer bd010bbd952447cb8aab0a8e79691a65: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
14:22:27.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:27.393575 27813 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
14:22:28.595 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:28.596 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:28.597 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=13, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=2774), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [0ms] delaying RPC due to: Illegal state: connection is terminated, [7ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [10ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [10ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [31ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [33ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [33ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [51ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [53ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [55ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [55ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [87ms] refreshing cache from master, [87ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [91ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [92ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [94ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [94ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [166ms] refreshing cache from master, [166ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [170ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [172ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [174ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [174ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [646ms] refreshing cache from master, [647ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [650ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [652ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [654ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [654ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [2766ms] refreshing cache from master, [2766ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [2772ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1513515716(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@330b7178 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@115228619 after retry RPC@459407456) -> (continuation of Deferred@1280272357 after retry RPC@1193615116) -> (continuation of Deferred@448833684 after retry RPC@1978800575) -> (continuation of Deferred@1257496567 after retry RPC@791222519) -> (continuation of Deferred@210381394 after retry RPC@787701242), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ad6eeee -> wakeup thread Time-limited test -> (continuation of Deferred@115228619 after retry RPC@459407456) -> (continuation of Deferred@1280272357 after retry RPC@1193615116) -> (continuation of Deferred@448833684 after retry RPC@1978800575) -> (continuation of Deferred@1257496567 after retry RPC@791222519) -> (continuation of Deferred@210381394 after retry RPC@787701242))) after lookup
14:22:28.597 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:28.599 [DEBUG - kudu-nio-6] (Connection.java:757) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] Successfully connected to peer
14:22:28.636 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0x869e23fd, L:/127.0.0.1:56886 - R:/127.25.218.131:44025] using TOKEN/null
14:22:28.699 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.218.130:39771
14:22:28.700 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:28.701696 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/logs
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.130:39771
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.130
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37055
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.130
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:29.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:29.106307 27836 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:29.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:29.106817 27836 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:29.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:29.107285 27836 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:29.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:29.143389 27836 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:29.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:29.144279 27836 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.130
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:29.187445 27836 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.130:39771
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.130
14:22:29.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37055
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.130
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/logs
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:29.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:29.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:29.189801 27836 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:29.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:29.192672 27836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:29.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:29.210229 27842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:29.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:29.966437 27489 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 -> Peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Couldn't send request to peer 34a6cb78ecdf4d7a85465851c4c3bb04. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:22:31.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:29.210533 27843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:31.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:30.612692 27841 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 27836
14:22:31.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:31.212550 27841 kernel_stack_watchdog.cc:198] Thread 27836 stuck at /home/jenkins-slave/workspace/build_and_test@2/src/kudu/util/thread.cc:642 for 399ms:
14:22:31.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:22:31.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:22:31.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:22:31.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:22:31.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:31.214368 27836 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.005s	user 0.725s	sys 1.256s
14:22:31.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:31.215072 27836 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.006s	user 0.725s	sys 1.256s
14:22:31.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:31.227870 27844 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2003 milliseconds
14:22:31.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:31.230350 27845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:31.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.230418 27836 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:22:31.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.232131 27836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:31.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.235329 27836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.236843 27836 hybrid_clock.cc:648] HybridClock initialized: now 1779114151236788 us; error 53 us; skew 500 ppm
14:22:31.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.237938 27836 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:31.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.246435 27836 webserver.cc:533] Webserver started at http://127.25.218.130:37055/ using document root <none> and password file <none>
14:22:31.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.248162 27836 fs_manager.cc:362] Metadata directory not provided
14:22:31.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.248564 27836 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:31.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.259665 27836 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.000s
14:22:31.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.266978 27862 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:31.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.268483 27836 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.004s	sys 0.003s
14:22:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.268832 27836 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34a6cb78ecdf4d7a85465851c4c3bb04"
14:22:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:07 on dist-test-slave-45s3"
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.271581 27836 fs_report.cc:389] FS layout report
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/data
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:31.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.328086 27836 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:31.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.329689 27836 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:31.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.330116 27836 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:31.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.333052 27836 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:31.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.339075 27869 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:22:31.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.349152 27836 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:22:31.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.349524 27836 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.001s	sys 0.001s
14:22:31.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.349867 27836 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:22:31.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.357398 27836 ts_tablet_manager.cc:616] Registered 1 tablets
14:22:31.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.357673 27836 ts_tablet_manager.cc:595] Time spent register tablets: real 0.008s	user 0.007s	sys 0.000s
14:22:31.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.358244 27869 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap starting.
14:22:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.433609 27869 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Log is configured to *not* fsync() on all Append() calls
14:22:31.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.532375 27869 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:22:31.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.533491 27869 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap complete.
14:22:31.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.535041 27869 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Time spent bootstrapping tablet: real 0.177s	user 0.149s	sys 0.020s
14:22:31.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.557665 27869 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:31.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.560488 27869 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a6cb78ecdf4d7a85465851c4c3bb04, State: Initialized, Role: FOLLOWER
14:22:31.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.561741 27869 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:31.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.567745 27869 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Time spent starting tablet: real 0.032s	user 0.027s	sys 0.008s
14:22:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.593258 27836 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.130:39771
14:22:31.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.593931 27977 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.130:39771 every 8 connection(s)
14:22:31.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.596341 27836 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:31.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.604025 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 27836
14:22:31.605 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:31.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.651722 27978 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:31.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.652247 27978 heartbeater.cc:461] Registering TS with master...
14:22:31.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.653751 27978 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:31.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.659801 26674 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:31.662 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:22:31.664 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.218.131:44025
14:22:31.664 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:22:31.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.663584 26674 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.130:55603
14:22:31.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.667205 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 27465
14:22:31.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.677793 27979 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:31.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.678272 27979 heartbeater.cc:461] Registering TS with master...
14:22:31.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.679159 27979 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:31.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.682418 27980 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:31.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.682808 27980 heartbeater.cc:461] Registering TS with master...
14:22:31.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.683561 27980 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:31.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.684250 26593 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:31.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:31.688407 26518 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:31.712 [DEBUG - kudu-nio-6] (Connection.java:239) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] handling channelInactive
14:22:31.712 [DEBUG - kudu-nio-6] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state READY due to: connection closed
14:22:31.714 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:31.715 [DEBUG - Time-limited test] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:31.716 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.716 [DEBUG - kudu-nio-7] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.717 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.734 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:31.734 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:31.735 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@793256345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1b60a507 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@546de551 -> wakeup thread Time-limited test)) after lookup
14:22:31.736 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:31.737 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.737 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.737 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.753 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:31.754 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:31.754 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [23ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@793256345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1b60a507 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1173710578 after retry RPC@1013411637), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@546de551 -> wakeup thread Time-limited test -> (continuation of Deferred@1173710578 after retry RPC@1013411637))) after lookup
14:22:31.755 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:31.756 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.756 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.756 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.792 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:31.792 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:31.793 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=78), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [23ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [40ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [42ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [42ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [78ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@793256345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1b60a507 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1173710578 after retry RPC@1013411637) -> (continuation of Deferred@142653716 after retry RPC@261714422), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@546de551 -> wakeup thread Time-limited test -> (continuation of Deferred@1173710578 after retry RPC@1013411637) -> (continuation of Deferred@142653716 after retry RPC@261714422))) after lookup
14:22:31.793 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:31.794 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.794 [DEBUG - kudu-nio-2] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.795 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.893 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:31.894 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:31.895 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=180), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [23ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [40ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [42ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [42ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [78ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [79ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [80ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [81ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [174ms] refreshing cache from master, [175ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [179ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@793256345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1b60a507 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1173710578 after retry RPC@1013411637) -> (continuation of Deferred@142653716 after retry RPC@261714422) -> (continuation of Deferred@1207813364 after retry RPC@61807420), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@546de551 -> wakeup thread Time-limited test -> (continuation of Deferred@1173710578 after retry RPC@1013411637) -> (continuation of Deferred@142653716 after retry RPC@261714422) -> (continuation of Deferred@1207813364 after retry RPC@61807420))) after lookup
14:22:31.896 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:31.898 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.898 [DEBUG - kudu-nio-3] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:31.898 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:32.414 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:32.415 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:32.415 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=701), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [23ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [40ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [42ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [42ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [78ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [79ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [80ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [81ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [174ms] refreshing cache from master, [175ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [179ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [182ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [184ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [184ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [695ms] refreshing cache from master, [695ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [699ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@793256345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1b60a507 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1173710578 after retry RPC@1013411637) -> (continuation of Deferred@142653716 after retry RPC@261714422) -> (continuation of Deferred@1207813364 after retry RPC@61807420) -> (continuation of Deferred@753793302 after retry RPC@1474326860), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@546de551 -> wakeup thread Time-limited test -> (continuation of Deferred@1173710578 after retry RPC@1013411637) -> (continuation of Deferred@142653716 after retry RPC@261714422) -> (continuation of Deferred@1207813364 after retry RPC@61807420) -> (continuation of Deferred@753793302 after retry RPC@1474326860))) after lookup
14:22:32.416 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:32.418 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:32.418 [DEBUG - kudu-nio-4] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:32.418 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:32.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.670572 27978 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:32.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.727200 27990 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bd010bbd952447cb8aab0a8e79691a65)
14:22:32.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.727649 27990 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:32.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.730212 27990 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:32.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:32.734193 27671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111)
14:22:32.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:32.739216 27671 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025): Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111)
14:22:32.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.743963 27932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "b575acd823514305a1d3e6f006cb011b" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" is_pre_election: true
14:22:32.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.744724 27932 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b575acd823514305a1d3e6f006cb011b in term 4.
14:22:32.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.746268 27670 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, b575acd823514305a1d3e6f006cb011b; no voters: bd010bbd952447cb8aab0a8e79691a65
14:22:32.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.747051 27990 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 5 FOLLOWER]: Leader pre-election won for term 6
14:22:32.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.747381 27990 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 5 FOLLOWER]: Starting leader election (detected failure of leader bd010bbd952447cb8aab0a8e79691a65)
14:22:32.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.748062 27990 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 5 FOLLOWER]: Advancing to term 6
14:22:32.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.753520 27990 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:32.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.755594 27990 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 6 election: Requested vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:32.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.756659 27932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "b575acd823514305a1d3e6f006cb011b" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04"
14:22:32.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.757122 27932 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 4 FOLLOWER]: Advancing to term 6
14:22:32.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:32.759914 27671 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025): Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111)
14:22:32.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.766270 27932 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b575acd823514305a1d3e6f006cb011b in term 6.
14:22:32.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.767448 27670 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, b575acd823514305a1d3e6f006cb011b; no voters: bd010bbd952447cb8aab0a8e79691a65
14:22:32.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.768276 27990 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 6 FOLLOWER]: Leader election won for term 6
14:22:32.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.769901 27990 raft_consensus.cc:697] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 6 LEADER]: Becoming Leader. State: Replica: b575acd823514305a1d3e6f006cb011b, State: Running, Role: LEADER
14:22:32.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.770838 27990 consensus_queue.cc:237] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:32.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:32.780471 26674 catalog_manager.cc:5697] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b reported cstate change: term changed from 5 to 6, leader changed from bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131) to b575acd823514305a1d3e6f006cb011b (127.25.218.129). New cstate: current_term: 6 leader_uuid: "b575acd823514305a1d3e6f006cb011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } health_report { overall_health: UNKNOWN } } }
14:22:33.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:33.188956 27932 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 6 FOLLOWER]: Refusing update from remote peer b575acd823514305a1d3e6f006cb011b: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
14:22:33.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:33.190797 27990 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
14:22:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:33.210053 27671 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b -> Peer bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025): Couldn't send request to peer bd010bbd952447cb8aab0a8e79691a65. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:22:34.454 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:34.455 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:34.455 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2741), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [3ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [3ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [23ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [23ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [39ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [40ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [42ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [42ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [78ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [79ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [80ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [81ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [174ms] refreshing cache from master, [175ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [179ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [182ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [184ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [184ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [695ms] refreshing cache from master, [695ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [699ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [702ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [704ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [704ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [2735ms] refreshing cache from master, [2735ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [2740ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@793256345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1b60a507 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1173710578 after retry RPC@1013411637) -> (continuation of Deferred@142653716 after retry RPC@261714422) -> (continuation of Deferred@1207813364 after retry RPC@61807420) -> (continuation of Deferred@753793302 after retry RPC@1474326860) -> (continuation of Deferred@63383382 after retry RPC@1475955827), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@546de551 -> wakeup thread Time-limited test -> (continuation of Deferred@1173710578 after retry RPC@1013411637) -> (continuation of Deferred@142653716 after retry RPC@261714422) -> (continuation of Deferred@1207813364 after retry RPC@61807420) -> (continuation of Deferred@753793302 after retry RPC@1474326860) -> (continuation of Deferred@63383382 after retry RPC@1475955827))) after lookup
14:22:34.456 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:34.458 [DEBUG - kudu-nio-5] (Connection.java:757) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] Successfully connected to peer
14:22:34.487 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x88464355, L:/127.0.0.1:42182 - R:/127.25.218.129:33083] using TOKEN/null
14:22:34.567 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.218.131:44025
14:22:34.568 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:22:34.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:34.569850 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:34.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:34.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:34.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data
14:22:34.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:34.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:34.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:34.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:34.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:34.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/logs
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:34.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.131:44025
14:22:34.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.131
14:22:34.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:34.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39731
14:22:34.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.131
14:22:34.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:34.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:34.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:34.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:34.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:34.942996 28027 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:34.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:34.943485 28027 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:34.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:34.944204 28027 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:34.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:34.985714 28027 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:34.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:34.987051 28027 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.131
14:22:35.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:35.031145 28027 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:35.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.131:44025
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.131
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39731
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.131
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/logs
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:35.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:35.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:35.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:35.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:35.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:35.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:35.033160 28027 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:35.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:35.035075 28027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:35.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:35.051375 28036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:35.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:35.574388 27671 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b -> Peer bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025): Couldn't send request to peer bd010bbd952447cb8aab0a8e79691a65. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:22:36.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:36.453701 28035 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 28027
14:22:36.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:36.994572 28035 kernel_stack_watchdog.cc:198] Thread 28027 stuck at /home/jenkins-slave/workspace/build_and_test@2/src/kudu/util/thread.cc:642 for 399ms:
14:22:36.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:22:36.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:22:36.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:22:36.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:22:36.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:35.053992 28037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:36.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:36.995939 28027 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.945s	user 0.634s	sys 1.270s
14:22:36.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:36.996555 28027 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.945s	user 0.634s	sys 1.270s
14:22:36.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:36.998288 28038 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1943 milliseconds
14:22:37.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:36.999912 28027 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:22:37.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:36.999814 28039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:37.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.009429 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:37.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.012228 28027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:37.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.013680 28027 hybrid_clock.cc:648] HybridClock initialized: now 1779114157013634 us; error 53 us; skew 500 ppm
14:22:37.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.014582 28027 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:37.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.024986 28027 webserver.cc:533] Webserver started at http://127.25.218.131:39731/ using document root <none> and password file <none>
14:22:37.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.026167 28027 fs_manager.cc:362] Metadata directory not provided
14:22:37.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.026403 28027 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:37.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.036136 28027 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.006s	sys 0.001s
14:22:37.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.041885 28053 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:37.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.043524 28027 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.006s	sys 0.000s
14:22:37.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.044039 28027 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:37.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd010bbd952447cb8aab0a8e79691a65"
14:22:37.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:09 on dist-test-slave-45s3"
14:22:37.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.046840 28027 fs_report.cc:389] FS layout report
14:22:37.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:37.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:37.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:37.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/data
14:22:37.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:37.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:37.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:37.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:37.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:37.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:37.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:37.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:37.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:37.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:37.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:37.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.109122 28027 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:37.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.111110 28027 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:37.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.111757 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:37.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.114820 28027 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:37.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.121421 28060 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:22:37.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.131372 28027 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:22:37.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.131726 28027 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.001s	sys 0.000s
14:22:37.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.132056 28027 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:22:37.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.137907 28027 ts_tablet_manager.cc:616] Registered 1 tablets
14:22:37.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.138171 28027 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.004s	sys 0.000s
14:22:37.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.138516 28060 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Bootstrap starting.
14:22:37.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.208001 28060 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Log is configured to *not* fsync() on all Append() calls
14:22:37.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.309151 28060 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:22:37.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.310212 28060 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Bootstrap complete.
14:22:37.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.311815 28060 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Time spent bootstrapping tablet: real 0.174s	user 0.131s	sys 0.036s
14:22:37.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.335623 28060 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:37.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.339704 28060 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd010bbd952447cb8aab0a8e79691a65, State: Initialized, Role: FOLLOWER
14:22:37.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.340915 28060 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:37.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.345782 28060 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Time spent starting tablet: real 0.034s	user 0.032s	sys 0.000s
14:22:37.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.354161 28027 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.131:44025
14:22:37.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.354612 28168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.131:44025 every 8 connection(s)
14:22:37.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.357769 28027 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:37.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.366583 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 28027
14:22:37.368 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:37.410 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:22:37.413 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.218.129:33083
14:22:37.414 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:22:37.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.414836 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 27642
14:22:37.477 [DEBUG - kudu-nio-5] (Connection.java:239) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] handling channelInactive
14:22:37.477 [DEBUG - kudu-nio-5] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state READY due to: connection closed
14:22:37.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:37.477320 28179 negotiation.cc:336] Failed RPC negotiation. Trace:
14:22:37.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:22:37.406621 (+     0us) reactor.cc:678] Submitting negotiation task for server connection from 127.25.218.129:58777 (local address 127.25.218.131:44025)
14:22:37.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:22:37.453501 (+ 46880us) server_negotiation.cc:207] Beginning negotiation
14:22:37.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:22:37.453554 (+    53us) server_negotiation.cc:400] Waiting for connection header
14:22:37.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:22:37.453699 (+   145us) server_negotiation.cc:408] Connection header received
14:22:37.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:22:37.454404 (+   705us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
14:22:37.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:22:37.454471 (+    67us) server_negotiation.cc:462] Received NEGOTIATE request from client
14:22:37.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:22:37.469953 (+ 15482us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
14:22:37.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:22:37.470276 (+   323us) server_negotiation.cc:227] Negotiated authn=SASL
14:22:37.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:22:37.475298 (+  5022us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.218.129:58777: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
14:22:37.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":34313,"thread_start_us":45831,"threads_started":1}
14:22:37.492 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:37.494 [DEBUG - Time-limited test] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:37.495 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.496 [DEBUG - kudu-nio-6] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.496 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.510478 28172 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:37.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.511103 28172 heartbeater.cc:461] Registering TS with master...
14:22:37.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.512678 28172 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:37.513 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:37.513 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:37.514 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [3ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@476998217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1e93763 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d03d35 -> wakeup thread Time-limited test)) after lookup
14:22:37.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.514000 28169 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:37.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.514400 28169 heartbeater.cc:461] Registering TS with master...
14:22:37.514 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:37.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.515385 28169 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:37.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.517731 26519 ts_manager.cc:194] Re-registered known tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:37.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.518321 28170 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:37.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.518671 28170 heartbeater.cc:461] Registering TS with master...
14:22:37.519 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.519 [DEBUG - kudu-nio-7] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.519 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.519521 26673 ts_manager.cc:194] Re-registered known tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:37.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.519657 28170 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:37.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.522868 26673 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.131:52321
14:22:37.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:37.523849 26594 ts_manager.cc:194] Re-registered known tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:37.533 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:37.533 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:37.534 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [3ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [26ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [26ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@476998217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1e93763 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d03d35 -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982))) after lookup
14:22:37.535 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:37.536 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.536 [DEBUG - kudu-nio-0] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.537 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.573 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:37.574 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:37.574 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=81), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [3ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [26ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [26ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [41ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [43ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [44ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [80ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@476998217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1e93763 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982) -> (continuation of Deferred@870977913 after retry RPC@175539459), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d03d35 -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982) -> (continuation of Deferred@870977913 after retry RPC@175539459))) after lookup
14:22:37.574 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:37.576 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.576 [DEBUG - kudu-nio-1] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.576 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.613 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:37.613 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:37.613 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=120), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [3ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [26ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [26ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [41ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [43ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [44ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [80ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [81ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [83ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [83ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [115ms] refreshing cache from master, [115ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [120ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@476998217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1e93763 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982) -> (continuation of Deferred@870977913 after retry RPC@175539459) -> (continuation of Deferred@1792430079 after retry RPC@1412862267), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d03d35 -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982) -> (continuation of Deferred@870977913 after retry RPC@175539459) -> (continuation of Deferred@1792430079 after retry RPC@1412862267))) after lookup
14:22:37.614 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:37.615 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.616 [DEBUG - kudu-nio-2] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.616 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.773 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:37.774 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:37.774 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=281), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [3ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [26ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [26ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [41ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [43ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [44ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [80ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [81ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [83ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [83ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [115ms] refreshing cache from master, [115ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [120ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [121ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [123ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [123ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [275ms] refreshing cache from master, [276ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [280ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@476998217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1e93763 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982) -> (continuation of Deferred@870977913 after retry RPC@175539459) -> (continuation of Deferred@1792430079 after retry RPC@1412862267) -> (continuation of Deferred@2121138336 after retry RPC@506587423), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d03d35 -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982) -> (continuation of Deferred@870977913 after retry RPC@175539459) -> (continuation of Deferred@1792430079 after retry RPC@1412862267) -> (continuation of Deferred@2121138336 after retry RPC@506587423))) after lookup
14:22:37.775 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:37.777 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.777 [DEBUG - kudu-nio-3] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.777 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.873 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:37.874 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:37.874 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=381), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [3ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [26ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [26ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [41ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [43ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [44ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [80ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [81ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [83ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [83ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [115ms] refreshing cache from master, [115ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [120ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [121ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [123ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [123ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [275ms] refreshing cache from master, [276ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [280ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [282ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [284ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [284ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [375ms] refreshing cache from master, [375ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [380ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@476998217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1e93763 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982) -> (continuation of Deferred@870977913 after retry RPC@175539459) -> (continuation of Deferred@1792430079 after retry RPC@1412862267) -> (continuation of Deferred@2121138336 after retry RPC@506587423) -> (continuation of Deferred@2009052730 after retry RPC@1717027011), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d03d35 -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982) -> (continuation of Deferred@870977913 after retry RPC@175539459) -> (continuation of Deferred@1792430079 after retry RPC@1412862267) -> (continuation of Deferred@2121138336 after retry RPC@506587423) -> (continuation of Deferred@2009052730 after retry RPC@1717027011))) after lookup
14:22:37.875 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:37.877 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.877 [DEBUG - kudu-nio-4] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:37.877 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.528582 28169 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:38.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.567562 28182 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b575acd823514305a1d3e6f006cb011b)
14:22:38.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.568105 28182 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:38.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.570456 28182 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:38.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:38.573841 27866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111)
14:22:38.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:38.577728 27866 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111)
14:22:38.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.583760 28123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "bd010bbd952447cb8aab0a8e79691a65" is_pre_election: true
14:22:38.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.584751 28123 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a6cb78ecdf4d7a85465851c4c3bb04 in term 5.
14:22:38.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.586164 27866 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, bd010bbd952447cb8aab0a8e79691a65; no voters: b575acd823514305a1d3e6f006cb011b
14:22:38.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.586944 28182 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 6 FOLLOWER]: Leader pre-election won for term 7
14:22:38.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.587296 28182 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b575acd823514305a1d3e6f006cb011b)
14:22:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.587680 28182 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 6 FOLLOWER]: Advancing to term 7
14:22:38.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.594792 28182 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:38.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.597092 28182 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 7 election: Requested vote from peers b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:38.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.598964 28123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "bd010bbd952447cb8aab0a8e79691a65"
14:22:38.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.599604 28123 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 5 FOLLOWER]: Advancing to term 7
14:22:38.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:38.602406 27866 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111)
14:22:38.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.610350 28123 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a6cb78ecdf4d7a85465851c4c3bb04 in term 7.
14:22:38.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.611863 27866 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, bd010bbd952447cb8aab0a8e79691a65; no voters: b575acd823514305a1d3e6f006cb011b
14:22:38.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.612625 28182 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 7 FOLLOWER]: Leader election won for term 7
14:22:38.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.614394 28182 raft_consensus.cc:697] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 7 LEADER]: Becoming Leader. State: Replica: 34a6cb78ecdf4d7a85465851c4c3bb04, State: Running, Role: LEADER
14:22:38.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.615435 28182 consensus_queue.cc:237] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:38.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:38.625639 26673 catalog_manager.cc:5697] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 reported cstate change: term changed from 6 to 7, leader changed from b575acd823514305a1d3e6f006cb011b (127.25.218.129) to 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130). New cstate: current_term: 7 leader_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } health_report { overall_health: UNKNOWN } } }
14:22:39.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:39.118310 27866 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 -> Peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Couldn't send request to peer b575acd823514305a1d3e6f006cb011b. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:22:39.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:39.154492 28123 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 7 FOLLOWER]: Refusing update from remote peer 34a6cb78ecdf4d7a85465851c4c3bb04: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
14:22:39.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:39.156497 28182 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.001s
14:22:40.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:40.060070 26705 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac720 after lost signal to thread 26636
14:22:40.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:40.061362 26705 debug-util.cc:398] Leaking SignalData structure 0x7b08000adf00 after lost signal to thread 26708
14:22:40.314 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:40.314 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:40.315 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2821), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [3ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [3ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [20ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [21ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [26ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [26ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [40ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [41ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [43ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [44ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [80ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [81ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [83ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [83ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [115ms] refreshing cache from master, [115ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [120ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [121ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [123ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [123ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [275ms] refreshing cache from master, [276ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [280ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [282ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [284ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [284ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [375ms] refreshing cache from master, [375ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [380ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [382ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [384ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [384ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [2816ms] refreshing cache from master, [2816ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [2821ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@476998217(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@1e93763 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982) -> (continuation of Deferred@870977913 after retry RPC@175539459) -> (continuation of Deferred@1792430079 after retry RPC@1412862267) -> (continuation of Deferred@2121138336 after retry RPC@506587423) -> (continuation of Deferred@2009052730 after retry RPC@1717027011) -> (continuation of Deferred@853375281 after retry RPC@1783948913), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d03d35 -> wakeup thread Time-limited test -> (continuation of Deferred@1065194889 after retry RPC@772753982) -> (continuation of Deferred@870977913 after retry RPC@175539459) -> (continuation of Deferred@1792430079 after retry RPC@1412862267) -> (continuation of Deferred@2121138336 after retry RPC@506587423) -> (continuation of Deferred@2009052730 after retry RPC@1717027011) -> (continuation of Deferred@853375281 after retry RPC@1783948913))) after lookup
14:22:40.316 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:40.317 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] Successfully connected to peer
14:22:40.339 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x374bbb26, L:/127.0.0.1:60676 - R:/127.25.218.130:39771] using TOKEN/null
14:22:40.401 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.218.129:33083
14:22:40.401 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:22:40.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:40.403129 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:40.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:40.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:40.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data
14:22:40.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:40.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:40.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/logs
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.129:33083
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.129
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40225
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.129
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:40.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:40.760469 28222 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:40.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:40.761152 28222 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:40.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:40.761719 28222 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:40.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:40.799230 28222 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:40.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:40.800237 28222 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.129
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:40.839524 28222 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.129:33083
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.129
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40225
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:40.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.129
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/logs
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:40.841214 28222 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:40.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:40.842832 28222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:40.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:40.857717 28228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:41.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:41.619827 27866 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 -> Peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Couldn't send request to peer b575acd823514305a1d3e6f006cb011b. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:22:42.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:40.859289 28229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:42.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:40.861481 28231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:42.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:42.291458 28230 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1429 milliseconds
14:22:42.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.291913 28222 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:22:42.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.293218 28222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:42.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.295748 28222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:42.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.297195 28222 hybrid_clock.cc:648] HybridClock initialized: now 1779114162297150 us; error 65 us; skew 500 ppm
14:22:42.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.298053 28222 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:42.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.305240 28222 webserver.cc:533] Webserver started at http://127.25.218.129:40225/ using document root <none> and password file <none>
14:22:42.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.306373 28222 fs_manager.cc:362] Metadata directory not provided
14:22:42.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.306602 28222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:42.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.315652 28222 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.006s	sys 0.001s
14:22:42.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.321427 28241 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:42.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.323113 28222 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.000s
14:22:42.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.323436 28222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:42.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b575acd823514305a1d3e6f006cb011b"
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:05 on dist-test-slave-45s3"
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.325700 28222 fs_report.cc:389] FS layout report
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/data
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:42.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:42.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.378664 28222 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:42.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.380537 28222 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:42.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.381129 28222 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.384035 28222 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:42.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.390692 28248 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:22:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.400960 28222 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:22:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.401216 28222 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.002s	sys 0.001s
14:22:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.401463 28222 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:22:42.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.406966 28222 ts_tablet_manager.cc:616] Registered 1 tablets
14:22:42.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.407191 28222 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.003s	sys 0.000s
14:22:42.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.408001 28248 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Bootstrap starting.
14:22:42.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.485411 28248 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Log is configured to *not* fsync() on all Append() calls
14:22:42.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.607056 28248 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:22:42.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.608103 28248 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Bootstrap complete.
14:22:42.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.609920 28248 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Time spent bootstrapping tablet: real 0.203s	user 0.150s	sys 0.046s
14:22:42.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.628590 28222 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.129:33083
14:22:42.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.629257 28355 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.129:33083 every 8 connection(s)
14:22:42.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.631983 28222 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:42.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.632910 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 28222
14:22:42.634 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:42.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.635015 28248 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:42.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.638628 28248 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: b575acd823514305a1d3e6f006cb011b, State: Initialized, Role: FOLLOWER
14:22:42.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.639962 28248 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:42.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.647949 28248 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Time spent starting tablet: real 0.038s	user 0.038s	sys 0.000s
14:22:42.685 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:22:42.686 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.218.130:39771
14:22:42.687 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:22:42.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.688488 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 27836
14:22:42.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.712458 28357 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:42.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.712949 28357 heartbeater.cc:461] Registering TS with master...
14:22:42.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.714408 28357 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:42.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.716238 28358 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:42.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.718073 28358 heartbeater.cc:461] Registering TS with master...
14:22:42.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.719132 28358 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:42.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.721675 26593 ts_manager.cc:194] Re-registered known tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:42.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.723122 26518 ts_manager.cc:194] Re-registered known tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:42.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.725610 28356 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:42.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.726091 28356 heartbeater.cc:461] Registering TS with master...
14:22:42.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.727283 28356 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:42.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.730279 26674 ts_manager.cc:194] Re-registered known tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:42.733 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] handling channelInactive
14:22:42.733 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state READY due to: connection closed
14:22:42.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.733256 26674 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.129:59289
14:22:42.735 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:42.736 [DEBUG - Time-limited test] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:42.737 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.737 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.737 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:42.737329 28356 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:42.752 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:42.753 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:42.753 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [2ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@409644931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55c0b591 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ce05e21 -> wakeup thread Time-limited test)) after lookup
14:22:42.753 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:42.755 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.755 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.755 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.773 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:42.774 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:42.774 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [2ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [20ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [20ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [38ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@409644931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55c0b591 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ce05e21 -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109))) after lookup
14:22:42.775 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:42.776 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.776 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.776 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.813 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:42.813 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:42.813 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=78), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [2ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [20ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [20ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [38ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [39ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [41ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [41ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [77ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@409644931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55c0b591 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109) -> (continuation of Deferred@466710446 after retry RPC@2061451835), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ce05e21 -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109) -> (continuation of Deferred@466710446 after retry RPC@2061451835))) after lookup
14:22:42.814 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:42.815 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.815 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.815 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.852 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:42.852 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:42.853 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=117), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [2ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [20ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [20ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [38ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [39ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [41ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [41ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [77ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [79ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [80ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [80ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [117ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@409644931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55c0b591 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109) -> (continuation of Deferred@466710446 after retry RPC@2061451835) -> (continuation of Deferred@1338185894 after retry RPC@1326274047), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ce05e21 -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109) -> (continuation of Deferred@466710446 after retry RPC@2061451835) -> (continuation of Deferred@1338185894 after retry RPC@1326274047))) after lookup
14:22:42.853 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:42.854 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.854 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:42.854 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:43.294 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:43.294 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:43.294 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=559), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [2ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [20ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [20ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [38ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [39ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [41ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [41ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [77ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [79ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [80ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [80ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [117ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [118ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [119ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [119ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [553ms] refreshing cache from master, [554ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [558ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@409644931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55c0b591 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109) -> (continuation of Deferred@466710446 after retry RPC@2061451835) -> (continuation of Deferred@1338185894 after retry RPC@1326274047) -> (continuation of Deferred@218849186 after retry RPC@671891793), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ce05e21 -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109) -> (continuation of Deferred@466710446 after retry RPC@2061451835) -> (continuation of Deferred@1338185894 after retry RPC@1326274047) -> (continuation of Deferred@218849186 after retry RPC@671891793))) after lookup
14:22:43.295 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:43.296 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:43.296 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:43.297 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:43.992 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:43.993 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:43.993 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1258), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [2ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [20ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [20ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [38ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [39ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [41ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [41ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [77ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [79ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [80ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [80ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [117ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [118ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [119ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [119ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [553ms] refreshing cache from master, [554ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [558ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [560ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [562ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [562ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [1253ms] refreshing cache from master, [1254ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [1257ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@409644931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55c0b591 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109) -> (continuation of Deferred@466710446 after retry RPC@2061451835) -> (continuation of Deferred@1338185894 after retry RPC@1326274047) -> (continuation of Deferred@218849186 after retry RPC@671891793) -> (continuation of Deferred@155045314 after retry RPC@1131479933), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ce05e21 -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109) -> (continuation of Deferred@466710446 after retry RPC@2061451835) -> (continuation of Deferred@1338185894 after retry RPC@1326274047) -> (continuation of Deferred@218849186 after retry RPC@671891793) -> (continuation of Deferred@155045314 after retry RPC@1131479933))) after lookup
14:22:43.994 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:43.994 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:43.995 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:43.995 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.016377 28368 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 34a6cb78ecdf4d7a85465851c4c3bb04)
14:22:44.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.016815 28368 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:44.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.019186 28368 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:44.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:44.023484 28056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:22:44.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:44.029501 28056 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:22:44.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.033483 28311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "bd010bbd952447cb8aab0a8e79691a65" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "b575acd823514305a1d3e6f006cb011b" is_pre_election: true
14:22:44.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.034323 28311 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd010bbd952447cb8aab0a8e79691a65 in term 6.
14:22:44.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.036197 28057 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b575acd823514305a1d3e6f006cb011b, bd010bbd952447cb8aab0a8e79691a65; no voters: 34a6cb78ecdf4d7a85465851c4c3bb04
14:22:44.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.037155 28368 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 7 FOLLOWER]: Leader pre-election won for term 8
14:22:44.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.037487 28368 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 34a6cb78ecdf4d7a85465851c4c3bb04)
14:22:44.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.037732 28368 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 7 FOLLOWER]: Advancing to term 8
14:22:44.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.042533 28368 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:44.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.044245 28368 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 8 election: Requested vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.045598 28311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "bd010bbd952447cb8aab0a8e79691a65" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "b575acd823514305a1d3e6f006cb011b"
14:22:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.046281 28311 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 6 FOLLOWER]: Advancing to term 8
14:22:44.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:44.048117 28056 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:22:44.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.056044 28311 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd010bbd952447cb8aab0a8e79691a65 in term 8.
14:22:44.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.057291 28057 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b575acd823514305a1d3e6f006cb011b, bd010bbd952447cb8aab0a8e79691a65; no voters: 34a6cb78ecdf4d7a85465851c4c3bb04
14:22:44.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.058169 28368 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 8 FOLLOWER]: Leader election won for term 8
14:22:44.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.060161 28368 raft_consensus.cc:697] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 8 LEADER]: Becoming Leader. State: Replica: bd010bbd952447cb8aab0a8e79691a65, State: Running, Role: LEADER
14:22:44.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.061399 28368 consensus_queue.cc:237] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:44.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.072345 26674 catalog_manager.cc:5697] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 reported cstate change: term changed from 7 to 8, leader changed from 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130) to bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131). New cstate: current_term: 8 leader_uuid: "bd010bbd952447cb8aab0a8e79691a65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } health_report { overall_health: HEALTHY } } }
14:22:44.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.469133 28311 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 8 FOLLOWER]: Refusing update from remote peer bd010bbd952447cb8aab0a8e79691a65: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
14:22:44.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:44.470577 28368 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
14:22:44.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:44.510147 28056 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 -> Peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Couldn't send request to peer 34a6cb78ecdf4d7a85465851c4c3bb04. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:22:46.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:46.885211 28056 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 -> Peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Couldn't send request to peer 34a6cb78ecdf4d7a85465851c4c3bb04. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:22:47.994 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:47.994 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:47.995 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=5259), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [2ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [2ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [17ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [18ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [20ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [20ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [38ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [39ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [41ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [41ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [77ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [79ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [80ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [80ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [117ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [118ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [119ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [119ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [553ms] refreshing cache from master, [554ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [558ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [560ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [562ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [562ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [1253ms] refreshing cache from master, [1254ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [1257ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [1258ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [1260ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [1260ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [5254ms] refreshing cache from master, [5254ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [5258ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@409644931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55c0b591 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109) -> (continuation of Deferred@466710446 after retry RPC@2061451835) -> (continuation of Deferred@1338185894 after retry RPC@1326274047) -> (continuation of Deferred@218849186 after retry RPC@671891793) -> (continuation of Deferred@155045314 after retry RPC@1131479933) -> (continuation of Deferred@734066960 after retry RPC@1617001982), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ce05e21 -> wakeup thread Time-limited test -> (continuation of Deferred@1783130259 after retry RPC@1683432109) -> (continuation of Deferred@466710446 after retry RPC@2061451835) -> (continuation of Deferred@1338185894 after retry RPC@1326274047) -> (continuation of Deferred@218849186 after retry RPC@671891793) -> (continuation of Deferred@155045314 after retry RPC@1131479933) -> (continuation of Deferred@734066960 after retry RPC@1617001982))) after lookup
14:22:47.995 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:47.996 [DEBUG - kudu-nio-5] (Connection.java:757) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] Successfully connected to peer
14:22:48.022 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0xac01bdd6, L:/127.0.0.1:52024 - R:/127.25.218.131:44025] using TOKEN/null
14:22:48.099 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.218.130:39771
14:22:48.100 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:22:48.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:48.103056 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:48.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:48.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:48.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data
14:22:48.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:48.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:48.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:48.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:48.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:48.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:48.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/logs
14:22:48.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:48.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:48.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:48.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:48.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:48.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.130:39771
14:22:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.130
14:22:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37055
14:22:48.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.130
14:22:48.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:48.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:48.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:48.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:48.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:48.498574 28406 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:48.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:48.499279 28406 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:48.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:48.499976 28406 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:48.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:48.542109 28406 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:48.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:48.543327 28406 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.130
14:22:48.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:48.585134 28406 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:48.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:48.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:48.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:48.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:48.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data
14:22:48.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.130:39771
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.130
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37055
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.130
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/logs
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:48.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:48.586457 28406 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:48.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:48.588233 28406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:48.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:48.606014 28414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:49.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:49.220690 28056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111) [suppressed 11 similar messages]
14:22:49.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:49.224690 28056 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 -> Peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Couldn't send request to peer 34a6cb78ecdf4d7a85465851c4c3bb04. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
14:22:49.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:48.612888 28415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:49.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:48.615036 28406 server_base.cc:1061] running on GCE node
14:22:49.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:48.615873 28417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:49.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.931270 28406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:49.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.933821 28406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:49.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.935308 28406 hybrid_clock.cc:648] HybridClock initialized: now 1779114169935267 us; error 43 us; skew 500 ppm
14:22:49.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.936486 28406 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:49.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.943626 28406 webserver.cc:533] Webserver started at http://127.25.218.130:37055/ using document root <none> and password file <none>
14:22:49.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.944804 28406 fs_manager.cc:362] Metadata directory not provided
14:22:49.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.945075 28406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:49.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.955029 28406 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.005s	sys 0.000s
14:22:49.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.960805 28425 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:49.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.962400 28406 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.000s	sys 0.004s
14:22:49.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.962899 28406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:49.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34a6cb78ecdf4d7a85465851c4c3bb04"
14:22:49.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:07 on dist-test-slave-45s3"
14:22:49.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:49.965572 28406 fs_report.cc:389] FS layout report
14:22:49.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:49.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:49.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:22:49.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/data
14:22:49.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:49.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:49.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:49.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:49.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:49.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:49.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:49.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:49.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:49.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:49.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:50.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.033253 28406 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:50.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.035267 28406 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:50.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.036201 28406 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:50.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.039992 28406 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:50.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.048503 28432 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:22:50.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.058370 28406 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:22:50.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.058670 28406 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.000s	sys 0.004s
14:22:50.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.059008 28406 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:22:50.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.067610 28406 ts_tablet_manager.cc:616] Registered 1 tablets
14:22:50.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.068261 28406 ts_tablet_manager.cc:595] Time spent register tablets: real 0.009s	user 0.010s	sys 0.000s
14:22:50.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.068470 28432 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap starting.
14:22:50.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.165437 28432 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Log is configured to *not* fsync() on all Append() calls
14:22:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.284596 28432 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:22:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.285610 28432 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap complete.
14:22:50.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.287485 28432 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Time spent bootstrapping tablet: real 0.220s	user 0.164s	sys 0.046s
14:22:50.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.313024 28432 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:50.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.316465 28432 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a6cb78ecdf4d7a85465851c4c3bb04, State: Initialized, Role: FOLLOWER
14:22:50.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.318039 28432 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:50.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.322993 28432 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Time spent starting tablet: real 0.035s	user 0.030s	sys 0.002s
14:22:50.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.412146 28406 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.130:39771
14:22:50.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.412515 28543 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.130:39771 every 8 connection(s)
14:22:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.415536 28406 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:22:50.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.424630 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 28406
14:22:50.426 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:50.512 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:22:50.513 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.218.131:44025
14:22:50.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.511965 28546 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:50.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.512508 28546 heartbeater.cc:461] Registering TS with master...
14:22:50.513 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:22:50.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.513896 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 28027
14:22:50.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.513974 28546 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:50.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.520399 26519 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:50.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.521355 28544 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:50.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.521786 28544 heartbeater.cc:461] Registering TS with master...
14:22:50.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.522970 28544 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:50.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.526558 26674 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:50.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.529992 28545 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:50.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.530433 28545 heartbeater.cc:461] Registering TS with master...
14:22:50.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.531884 28545 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:50.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.536512 26593 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:22:50.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:50.536931 26674 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.130:54755
14:22:50.559 [DEBUG - kudu-nio-5] (Connection.java:239) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] handling channelInactive
14:22:50.559 [DEBUG - kudu-nio-5] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state READY due to: connection closed
14:22:50.560 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:50.561 [DEBUG - Time-limited test] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:50.563 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.563 [DEBUG - kudu-nio-6] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.563 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.573 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:50.573 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:50.574 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [2ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [12ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@37984242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@692694c9 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c47e879 -> wakeup thread Time-limited test)) after lookup
14:22:50.574 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:50.576 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.576 [DEBUG - kudu-nio-7] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.576 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.593 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:50.593 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:50.593 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [2ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [12ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [13ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [15ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [15ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [31ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@37984242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@692694c9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1049917377 after retry RPC@1833129034), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c47e879 -> wakeup thread Time-limited test -> (continuation of Deferred@1049917377 after retry RPC@1833129034))) after lookup
14:22:50.594 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:50.595 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.595 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.596 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.612 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:50.613 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:50.613 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [2ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [12ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [13ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [15ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [15ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [31ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [33ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [34ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [35ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [51ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@37984242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@692694c9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1049917377 after retry RPC@1833129034) -> (continuation of Deferred@1722209629 after retry RPC@1438611345), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c47e879 -> wakeup thread Time-limited test -> (continuation of Deferred@1049917377 after retry RPC@1833129034) -> (continuation of Deferred@1722209629 after retry RPC@1438611345))) after lookup
14:22:50.614 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:50.615 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.615 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.615 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.713 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:50.713 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:50.714 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=152), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [2ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [12ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [13ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [15ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [15ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [31ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [33ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [34ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [35ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [51ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [52ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [54ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [55ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [151ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@37984242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@692694c9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1049917377 after retry RPC@1833129034) -> (continuation of Deferred@1722209629 after retry RPC@1438611345) -> (continuation of Deferred@1172742421 after retry RPC@1808483783), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c47e879 -> wakeup thread Time-limited test -> (continuation of Deferred@1049917377 after retry RPC@1833129034) -> (continuation of Deferred@1722209629 after retry RPC@1438611345) -> (continuation of Deferred@1172742421 after retry RPC@1808483783))) after lookup
14:22:50.714 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:50.716 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.716 [DEBUG - kudu-nio-2] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:50.716 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:51.193 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:51.193 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:22:51.193 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=632), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [2ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [12ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [13ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [15ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [15ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [31ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [33ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [34ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [35ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [51ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [52ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [54ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [55ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [151ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [153ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [155ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [155ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [627ms] refreshing cache from master, [627ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [631ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@37984242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@692694c9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1049917377 after retry RPC@1833129034) -> (continuation of Deferred@1722209629 after retry RPC@1438611345) -> (continuation of Deferred@1172742421 after retry RPC@1808483783) -> (continuation of Deferred@1671404515 after retry RPC@1389043498), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c47e879 -> wakeup thread Time-limited test -> (continuation of Deferred@1049917377 after retry RPC@1833129034) -> (continuation of Deferred@1722209629 after retry RPC@1438611345) -> (continuation of Deferred@1172742421 after retry RPC@1808483783) -> (continuation of Deferred@1671404515 after retry RPC@1389043498))) after lookup
14:22:51.194 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:22:51.196 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:51.196 [DEBUG - kudu-nio-3] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state CONNECTING due to: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:51.196 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025
14:22:51.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.542733 28544 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:51.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.627911 28556 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bd010bbd952447cb8aab0a8e79691a65)
14:22:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.629146 28556 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:51.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.632508 28556 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:51.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:51.647979 28245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111)
14:22:51.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:51.653373 28245 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025): Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111)
14:22:51.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.656064 28498 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "b575acd823514305a1d3e6f006cb011b" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" is_pre_election: true
14:22:51.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.657001 28498 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b575acd823514305a1d3e6f006cb011b in term 7.
14:22:51.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.658648 28244 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, b575acd823514305a1d3e6f006cb011b; no voters: bd010bbd952447cb8aab0a8e79691a65
14:22:51.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.659638 28556 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 8 FOLLOWER]: Leader pre-election won for term 9
14:22:51.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.660041 28556 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 8 FOLLOWER]: Starting leader election (detected failure of leader bd010bbd952447cb8aab0a8e79691a65)
14:22:51.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.660275 28556 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 8 FOLLOWER]: Advancing to term 9
14:22:51.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.665941 28556 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:51.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.667852 28556 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 9 election: Requested vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:51.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.669239 28498 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "b575acd823514305a1d3e6f006cb011b" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04"
14:22:51.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.670083 28498 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 7 FOLLOWER]: Advancing to term 9
14:22:51.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:51.672524 28245 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025): Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111)
14:22:51.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.677862 28498 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b575acd823514305a1d3e6f006cb011b in term 9.
14:22:51.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.679010 28244 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, b575acd823514305a1d3e6f006cb011b; no voters: bd010bbd952447cb8aab0a8e79691a65
14:22:51.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.679967 28556 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 9 FOLLOWER]: Leader election won for term 9
14:22:51.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.681599 28556 raft_consensus.cc:697] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 9 LEADER]: Becoming Leader. State: Replica: b575acd823514305a1d3e6f006cb011b, State: Running, Role: LEADER
14:22:51.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.682621 28556 consensus_queue.cc:237] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:51.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.694307 26674 catalog_manager.cc:5697] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b reported cstate change: term changed from 8 to 9, leader changed from bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131) to b575acd823514305a1d3e6f006cb011b (127.25.218.129). New cstate: current_term: 9 leader_uuid: "b575acd823514305a1d3e6f006cb011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } health_report { overall_health: UNKNOWN } } }
14:22:51.874 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:51.874 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:51.875 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1314), Traces: [0ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [2ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [2ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [12ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [13ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [15ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [15ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [31ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [33ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [34ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [35ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [51ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [52ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [54ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [55ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [151ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [153ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [155ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [155ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [627ms] refreshing cache from master, [627ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [631ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [633ms] sending RPC to server bd010bbd952447cb8aab0a8e79691a65, [635ms] received response from server bd010bbd952447cb8aab0a8e79691a65: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [635ms] delaying RPC due to: Network error: Failed to connect to peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025): Connection refused: /127.25.218.131:44025, [1308ms] refreshing cache from master, [1308ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [1312ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@37984242(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@692694c9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1049917377 after retry RPC@1833129034) -> (continuation of Deferred@1722209629 after retry RPC@1438611345) -> (continuation of Deferred@1172742421 after retry RPC@1808483783) -> (continuation of Deferred@1671404515 after retry RPC@1389043498) -> (continuation of Deferred@1710310623 after retry RPC@1875457284), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c47e879 -> wakeup thread Time-limited test -> (continuation of Deferred@1049917377 after retry RPC@1833129034) -> (continuation of Deferred@1722209629 after retry RPC@1438611345) -> (continuation of Deferred@1172742421 after retry RPC@1808483783) -> (continuation of Deferred@1671404515 after retry RPC@1389043498) -> (continuation of Deferred@1710310623 after retry RPC@1875457284))) after lookup
14:22:51.876 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:51.878 [DEBUG - kudu-nio-4] (Connection.java:757) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] Successfully connected to peer
14:22:51.916 [DEBUG - kudu-nio-4] (Negotiator.java:919) Authenticated connection [id: 0xe093a09e, L:/127.0.0.1:47502 - R:/127.25.218.129:33083] using TOKEN/null
14:22:51.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.930027 28498 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 9 FOLLOWER]: Refusing update from remote peer b575acd823514305a1d3e6f006cb011b: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 34. (index mismatch)
14:22:51.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:51.931943 28561 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.001s
14:22:51.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:51.932832 28245 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b -> Peer bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025): Couldn't send request to peer bd010bbd952447cb8aab0a8e79691a65. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.131:44025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:22:52.008 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.218.131:44025
14:22:52.009 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:22:52.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:52.010277 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/logs
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:52.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:52.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:52.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.131:44025
14:22:52.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.131
14:22:52.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:52.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39731
14:22:52.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.131
14:22:52.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:52.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:52.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:52.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:52.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:52.398520 28584 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:52.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:52.399092 28584 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:52.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:52.399555 28584 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:52.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:52.439410 28584 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:52.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:52.441028 28584 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.131
14:22:52.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:52.481953 28584 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:52.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:52.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:52.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:52.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:52.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data
14:22:52.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:52.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:52.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:52.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.131:44025
14:22:52.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.131
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39731
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.131
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/logs
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:52.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:52.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:52.483814 28584 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:52.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:52.485790 28584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:52.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:52.500741 28592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:53.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:52.502223 28593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:53.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:53.848918 28595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:53.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:53.851054 28594 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1344 milliseconds
14:22:53.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.851152 28584 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:22:53.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.852385 28584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:53.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.855147 28584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:53.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.856680 28584 hybrid_clock.cc:648] HybridClock initialized: now 1779114173856606 us; error 81 us; skew 500 ppm
14:22:53.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.857501 28584 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:53.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.864226 28584 webserver.cc:533] Webserver started at http://127.25.218.131:39731/ using document root <none> and password file <none>
14:22:53.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.865219 28584 fs_manager.cc:362] Metadata directory not provided
14:22:53.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.865410 28584 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:53.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.874439 28584 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.001s	sys 0.004s
14:22:53.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.879604 28604 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:53.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.880975 28584 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.003s
14:22:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.881276 28584 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bd010bbd952447cb8aab0a8e79691a65"
14:22:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:09 on dist-test-slave-45s3"
14:22:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.883291 28584 fs_report.cc:389] FS layout report
14:22:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/wal
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/data
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:53.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.948334 28584 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:53.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.950055 28584 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:53.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.950555 28584 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:53.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.953635 28584 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:53.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.960484 28611 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:22:53.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.969624 28584 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:22:53.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.969939 28584 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
14:22:53.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.970261 28584 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:22:53.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.975291 28584 ts_tablet_manager.cc:616] Registered 1 tablets
14:22:53.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.975531 28584 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.002s	sys 0.001s
14:22:53.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:53.976055 28611 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Bootstrap starting.
14:22:54.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.063184 28611 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Log is configured to *not* fsync() on all Append() calls
14:22:54.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.176440 28584 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.131:44025
14:22:54.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.176590 28718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.131:44025 every 8 connection(s)
14:22:54.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.180116 28584 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-2/data/info.pb
14:22:54.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.183744 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 28584
14:22:54.184 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:54.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.206614 28611 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:22:54.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.207803 28611 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Bootstrap complete.
14:22:54.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.209777 28611 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Time spent bootstrapping tablet: real 0.234s	user 0.181s	sys 0.045s
14:22:54.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.234724 28611 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:54.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.238191 28611 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd010bbd952447cb8aab0a8e79691a65, State: Initialized, Role: FOLLOWER
14:22:54.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.239780 28611 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:54.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.250628 28611 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65: Time spent starting tablet: real 0.041s	user 0.036s	sys 0.003s
14:22:54.254 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:22:54.255 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.218.129:33083
14:22:54.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:22:54.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.256634 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 28222
14:22:54.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.273671 28719 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:54.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.274271 28719 heartbeater.cc:461] Registering TS with master...
14:22:54.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.275620 28719 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:54.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.279924 26674 ts_manager.cc:194] Re-registered known tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:54.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.282784 26674 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.131:36279
14:22:54.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.287434 28719 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.292702 28727 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.293056 28727 heartbeater.cc:461] Registering TS with master...
14:22:54.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.293687 28727 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:54.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.297776 26519 ts_manager.cc:194] Re-registered known tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:54.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.300128 28720 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:54.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.300398 28720 heartbeater.cc:461] Registering TS with master...
14:22:54.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.300951 28720 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:54.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:54.303681 26593 ts_manager.cc:194] Re-registered known tserver with Master: bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:54.305 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:54.306 [DEBUG - kudu-nio-4] (Connection.java:239) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] handling channelInactive
14:22:54.306 [DEBUG - kudu-nio-4] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state READY due to: connection closed
14:22:54.306 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: connection closed
14:22:54.312 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:54.312 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:54.313 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=7), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [1ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1930507876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55565c17 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3496c271 -> wakeup thread Time-limited test)) after lookup
14:22:54.313 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:54.314 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.314 [DEBUG - kudu-nio-5] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.315 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.332 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:54.333 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:54.333 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [1ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [8ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [10ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [10ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1930507876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55565c17 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2132061941 after retry RPC@2106818092), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3496c271 -> wakeup thread Time-limited test -> (continuation of Deferred@2132061941 after retry RPC@2106818092))) after lookup
14:22:54.334 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:54.335 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.335 [DEBUG - kudu-nio-6] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.335 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.372 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:54.372 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:54.373 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=67), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [1ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [8ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [10ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [10ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [28ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [30ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [30ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [63ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [67ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1930507876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55565c17 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2132061941 after retry RPC@2106818092) -> (continuation of Deferred@411872157 after retry RPC@2122896574), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3496c271 -> wakeup thread Time-limited test -> (continuation of Deferred@2132061941 after retry RPC@2106818092) -> (continuation of Deferred@411872157 after retry RPC@2122896574))) after lookup
14:22:54.373 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:54.374 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.374 [DEBUG - kudu-nio-7] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.375 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.452 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:54.453 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:54.453 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=147), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [1ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [8ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [10ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [10ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [28ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [30ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [30ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [63ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [67ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [68ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [70ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [70ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [143ms] refreshing cache from master, [144ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [147ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1930507876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55565c17 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2132061941 after retry RPC@2106818092) -> (continuation of Deferred@411872157 after retry RPC@2122896574) -> (continuation of Deferred@1630058118 after retry RPC@1497211705), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3496c271 -> wakeup thread Time-limited test -> (continuation of Deferred@2132061941 after retry RPC@2106818092) -> (continuation of Deferred@411872157 after retry RPC@2122896574) -> (continuation of Deferred@1630058118 after retry RPC@1497211705))) after lookup
14:22:54.454 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:54.455 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.455 [DEBUG - kudu-nio-0] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.455 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.873 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:54.873 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)[L],bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:54.873 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=567), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [1ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [8ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [10ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [10ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [28ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [30ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [30ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [63ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [67ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [68ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [70ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [70ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [143ms] refreshing cache from master, [144ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [147ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [148ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [150ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [150ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [563ms] refreshing cache from master, [564ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [567ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1930507876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55565c17 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2132061941 after retry RPC@2106818092) -> (continuation of Deferred@411872157 after retry RPC@2122896574) -> (continuation of Deferred@1630058118 after retry RPC@1497211705) -> (continuation of Deferred@1991877126 after retry RPC@1717347478), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3496c271 -> wakeup thread Time-limited test -> (continuation of Deferred@2132061941 after retry RPC@2106818092) -> (continuation of Deferred@411872157 after retry RPC@2122896574) -> (continuation of Deferred@1630058118 after retry RPC@1497211705) -> (continuation of Deferred@1991877126 after retry RPC@1717347478))) after lookup
14:22:54.874 [DEBUG - kudu-nio-0] (Connection.java:748) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] connecting to peer
14:22:54.875 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.875 [DEBUG - kudu-nio-1] (Connection.java:709) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] cleaning up while in state CONNECTING due to: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:54.875 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083
14:22:55.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.656324 28732 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b575acd823514305a1d3e6f006cb011b)
14:22:55.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.656736 28732 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:55.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.658864 28732 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:55.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:55.662182 28429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111)
14:22:55.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:55.665802 28429 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111)
14:22:55.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.671401 28674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "bd010bbd952447cb8aab0a8e79691a65" is_pre_election: true
14:22:55.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.672276 28674 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a6cb78ecdf4d7a85465851c4c3bb04 in term 8.
14:22:55.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.673578 28737 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:22:55.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.673934 28429 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, bd010bbd952447cb8aab0a8e79691a65; no voters: b575acd823514305a1d3e6f006cb011b
14:22:55.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.674059 28737 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:55.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.674543 28732 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 9 FOLLOWER]: Leader pre-election won for term 10
14:22:55.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.674866 28732 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 9 FOLLOWER]: Starting leader election (detected failure of leader b575acd823514305a1d3e6f006cb011b)
14:22:55.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.675105 28732 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 9 FOLLOWER]: Advancing to term 10
14:22:55.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.676666 28737 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:55.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.683200 28732 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:55.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.685856 28732 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 10 election: Requested vote from peers b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083), bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131:44025)
14:22:55.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.688239 28674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "bd010bbd952447cb8aab0a8e79691a65"
14:22:55.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:55.688305 28608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111)
14:22:55.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.688920 28674 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 8 FOLLOWER]: Advancing to term 10
14:22:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:55.691766 28429 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111)
14:22:55.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:55.695801 28608 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111)
14:22:55.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.698177 28498 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "bd010bbd952447cb8aab0a8e79691a65" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" is_pre_election: true
14:22:55.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.698371 28674 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a6cb78ecdf4d7a85465851c4c3bb04 in term 10.
14:22:55.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.698889 28498 raft_consensus.cc:2368] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bd010bbd952447cb8aab0a8e79691a65 for earlier term 9. Current term is 10.
14:22:55.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.699306 28429 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34a6cb78ecdf4d7a85465851c4c3bb04, bd010bbd952447cb8aab0a8e79691a65; no voters: b575acd823514305a1d3e6f006cb011b
14:22:55.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.700088 28732 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 10 FOLLOWER]: Leader election won for term 10
14:22:55.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.700265 28607 leader_election.cc:400] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771) with higher term. Message: Invalid argument: T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bd010bbd952447cb8aab0a8e79691a65 for earlier term 9. Current term is 10.
14:22:55.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.700459 28607 leader_election.cc:403] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
14:22:55.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.700923 28737 raft_consensus.cc:2749] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771) with higher term. Message: Invalid argument: T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bd010bbd952447cb8aab0a8e79691a65 for earlier term 9. Current term is 10.
14:22:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.702314 28732 raft_consensus.cc:697] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 10 LEADER]: Becoming Leader. State: Replica: 34a6cb78ecdf4d7a85465851c4c3bb04, State: Running, Role: LEADER
14:22:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.703302 28732 consensus_queue.cc:237] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:55.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.712788 26674 catalog_manager.cc:5697] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 reported cstate change: term changed from 9 to 10, leader changed from b575acd823514305a1d3e6f006cb011b (127.25.218.129) to 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130). New cstate: current_term: 10 leader_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } health_report { overall_health: UNKNOWN } } }
14:22:55.792 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:55.793 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:55.793 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1487), Traces: [0ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [1ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [7ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [8ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [10ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [10ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [27ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [28ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [30ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [30ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [63ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [67ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [68ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [70ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [70ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [143ms] refreshing cache from master, [144ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [147ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [148ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [150ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [150ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [563ms] refreshing cache from master, [564ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [567ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [569ms] sending RPC to server b575acd823514305a1d3e6f006cb011b, [570ms] received response from server b575acd823514305a1d3e6f006cb011b: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [570ms] delaying RPC due to: Network error: Failed to connect to peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083): Connection refused: /127.25.218.129:33083, [1483ms] refreshing cache from master, [1484ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [1487ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1930507876(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@55565c17 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2132061941 after retry RPC@2106818092) -> (continuation of Deferred@411872157 after retry RPC@2122896574) -> (continuation of Deferred@1630058118 after retry RPC@1497211705) -> (continuation of Deferred@1991877126 after retry RPC@1717347478) -> (continuation of Deferred@1808220424 after retry RPC@1748775154), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3496c271 -> wakeup thread Time-limited test -> (continuation of Deferred@2132061941 after retry RPC@2106818092) -> (continuation of Deferred@411872157 after retry RPC@2122896574) -> (continuation of Deferred@1630058118 after retry RPC@1497211705) -> (continuation of Deferred@1991877126 after retry RPC@1717347478) -> (continuation of Deferred@1808220424 after retry RPC@1748775154))) after lookup
14:22:55.794 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:55.794 [DEBUG - kudu-nio-2] (Connection.java:757) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] Successfully connected to peer
14:22:55.811 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0xc66f0a25, L:/127.0.0.1:56508 - R:/127.25.218.130:39771] using TOKEN/null
14:22:55.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.820354 28674 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 10 FOLLOWER]: Refusing update from remote peer 34a6cb78ecdf4d7a85465851c4c3bb04: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 38. (index mismatch)
14:22:55.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.822237 28742 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.001s
14:22:55.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:55.822913 28429 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 -> Peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Couldn't send request to peer b575acd823514305a1d3e6f006cb011b. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:22:55.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.846361 28755 mvcc.cc:204] Tried to move back new op lower bound from 7287251664144097280 to 7287251663688282112. Current Snapshot: MvccSnapshot[applied={T|T < 7287251664144097280}]
14:22:55.885 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.218.129:33083
14:22:55.886 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:55.887621 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/logs
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:22:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:22:55.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.129:33083
14:22:55.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.129
14:22:55.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:55.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40225
14:22:55.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.129
14:22:55.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:55.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:55.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:55.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:22:56.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:56.251658 28763 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:22:56.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:56.252251 28763 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:22:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:56.252888 28763 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:22:56.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:56.288328 28763 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:22:56.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:56.289366 28763 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.129
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:56.328639 28763 tablet_server_runner.cc:78] Tablet server non-default flags:
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.129:33083
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.129
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40225
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:22:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.129
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/logs
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:22:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:56.330427 28763 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:22:56.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:56.332473 28763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:22:56.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:56.346393 28770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:58.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:57.749881 28769 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 28763
14:22:58.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:58.243664 28769 kernel_stack_watchdog.cc:198] Thread 28763 stuck at /home/jenkins-slave/workspace/build_and_test@2/src/kudu/util/thread.cc:642 for 400ms:
14:22:58.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:22:58.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:22:58.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:22:58.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:22:58.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:56.346778 28771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:58.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:58.243974 28763 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.897s	user 0.699s	sys 1.126s
14:22:58.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:58.246255 28763 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.900s	user 0.699s	sys 1.126s
14:22:58.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:58.247200 28773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:22:58.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:58.260310 28772 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1911 milliseconds
14:22:58.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.261044 28763 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:22:58.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.263938 28763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:22:58.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.266647 28763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:22:58.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.268301 28763 hybrid_clock.cc:648] HybridClock initialized: now 1779114178268226 us; error 79 us; skew 500 ppm
14:22:58.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.269479 28763 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:22:58.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.276862 28763 webserver.cc:533] Webserver started at http://127.25.218.129:40225/ using document root <none> and password file <none>
14:22:58.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.277765 28763 fs_manager.cc:362] Metadata directory not provided
14:22:58.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.277961 28763 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:22:58.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.285967 28763 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.000s	sys 0.004s
14:22:58.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.290905 28783 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:22:58.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.292300 28763 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
14:22:58.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.292657 28763 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:58.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b575acd823514305a1d3e6f006cb011b"
14:22:58.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:05 on dist-test-slave-45s3"
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.294652 28763 fs_report.cc:389] FS layout report
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/wal
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/data
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:22:58.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:22:58.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:22:58.343636 28429 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 -> Peer b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083): Couldn't send request to peer b575acd823514305a1d3e6f006cb011b. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.129:33083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:22:58.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.348469 28763 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:22:58.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.349931 28763 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:22:58.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.350332 28763 kserver.cc:163] Server-wide thread pool size limit: 3276
14:22:58.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.353011 28763 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:22:58.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.358932 28790 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:22:58.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.367305 28763 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:22:58.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.367630 28763 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
14:22:58.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.368016 28763 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:22:58.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.372851 28763 ts_tablet_manager.cc:616] Registered 1 tablets
14:22:58.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.373068 28763 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
14:22:58.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.373593 28790 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Bootstrap starting.
14:22:58.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.456074 28790 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Log is configured to *not* fsync() on all Append() calls
14:22:58.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.556030 28763 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.129:33083
14:22:58.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.556195 28898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.129:33083 every 8 connection(s)
14:22:58.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.559659 28763 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-0/data/info.pb
14:22:58.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.560952 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 28763
14:22:58.561 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:58.602 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:22:58.607 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.25.218.130:39771
14:22:58.609 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:22:58.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.609589 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 28406
14:22:58.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.617230 28790 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:22:58.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.625195 28790 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Bootstrap complete.
14:22:58.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.628688 28790 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Time spent bootstrapping tablet: real 0.256s	user 0.194s	sys 0.047s
14:22:58.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.640545 28900 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:22:58.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.641059 28900 heartbeater.cc:461] Registering TS with master...
14:22:58.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.642369 28900 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:22:58.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.647028 26593 ts_manager.cc:194] Re-registered known tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:58.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.647962 28899 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:22:58.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.648376 28899 heartbeater.cc:461] Registering TS with master...
14:22:58.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.649315 28904 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:22:58.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.649513 28899 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:22:58.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.649698 28904 heartbeater.cc:461] Registering TS with master...
14:22:58.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.650995 28904 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:22:58.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.653344 26673 ts_manager.cc:194] Re-registered known tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:58.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.654635 26518 ts_manager.cc:194] Re-registered known tserver with Master: b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:22:58.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.657157 26673 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.129:49155
14:22:58.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.657217 28790 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:58.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.660483 28790 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: b575acd823514305a1d3e6f006cb011b, State: Initialized, Role: FOLLOWER
14:22:58.660 [DEBUG - kudu-nio-2] (Connection.java:239) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] handling channelInactive
14:22:58.661 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state READY due to: connection closed
14:22:58.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.661598 28790 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:22:58.662 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:22:58.663 [DEBUG - Time-limited test] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:58.664 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.664 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.665 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.665779 28790 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b: Time spent starting tablet: real 0.037s	user 0.023s	sys 0.005s
14:22:58.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:22:58.668195 28899 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:22:58.674 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:58.674 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:58.674 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [3ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [3ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [11ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1976998391(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@7ce48c1f -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@35018e6f -> wakeup thread Time-limited test)) after lookup
14:22:58.674 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:58.676 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.676 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.676 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.692 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:58.692 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:58.693 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [3ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [3ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [11ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [12ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [14ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [14ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1976998391(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@7ce48c1f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@35018e6f -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434))) after lookup
14:22:58.693 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:58.694 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.694 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.694 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.732 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:58.732 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:58.733 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [3ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [3ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [11ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [12ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [14ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [14ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [31ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [32ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [32ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [70ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1976998391(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@7ce48c1f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434) -> (continuation of Deferred@1045002411 after retry RPC@385238341), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@35018e6f -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434) -> (continuation of Deferred@1045002411 after retry RPC@385238341))) after lookup
14:22:58.734 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:58.735 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.735 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.735 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.852 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:58.852 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:58.853 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=190), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [3ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [3ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [11ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [12ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [14ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [14ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [31ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [32ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [32ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [70ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [71ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [73ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [73ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [190ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1976998391(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@7ce48c1f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434) -> (continuation of Deferred@1045002411 after retry RPC@385238341) -> (continuation of Deferred@2125812153 after retry RPC@1515056967), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@35018e6f -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434) -> (continuation of Deferred@1045002411 after retry RPC@385238341) -> (continuation of Deferred@2125812153 after retry RPC@1515056967))) after lookup
14:22:58.853 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:58.854 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.854 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:58.855 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:59.073 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:59.073 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:59.074 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=411), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [3ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [3ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [11ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [12ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [14ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [14ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [31ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [32ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [32ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [70ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [71ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [73ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [73ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [190ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [191ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [193ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [193ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [406ms] refreshing cache from master, [406ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [411ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1976998391(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@7ce48c1f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434) -> (continuation of Deferred@1045002411 after retry RPC@385238341) -> (continuation of Deferred@2125812153 after retry RPC@1515056967) -> (continuation of Deferred@1464178828 after retry RPC@1648849652), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@35018e6f -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434) -> (continuation of Deferred@1045002411 after retry RPC@385238341) -> (continuation of Deferred@2125812153 after retry RPC@1515056967) -> (continuation of Deferred@1464178828 after retry RPC@1648849652))) after lookup
14:22:59.074 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:59.075 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:59.076 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:59.076 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:59.252 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:22:59.252 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)[L],b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)]}]
14:22:59.253 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=590), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [3ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [3ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [11ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [12ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [14ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [14ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [31ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [32ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [32ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [70ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [71ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [73ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [73ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [190ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [191ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [193ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [193ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [406ms] refreshing cache from master, [406ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [411ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [412ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [414ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [414ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [586ms] refreshing cache from master, [586ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [590ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1976998391(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@7ce48c1f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434) -> (continuation of Deferred@1045002411 after retry RPC@385238341) -> (continuation of Deferred@2125812153 after retry RPC@1515056967) -> (continuation of Deferred@1464178828 after retry RPC@1648849652) -> (continuation of Deferred@205662099 after retry RPC@517478150), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@35018e6f -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434) -> (continuation of Deferred@1045002411 after retry RPC@385238341) -> (continuation of Deferred@2125812153 after retry RPC@1515056967) -> (continuation of Deferred@1464178828 after retry RPC@1648849652) -> (continuation of Deferred@205662099 after retry RPC@517478150))) after lookup
14:22:59.253 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] connecting to peer
14:22:59.254 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:59.254 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] cleaning up while in state CONNECTING due to: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:22:59.255 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771) for tablet 747213ca655547928c53ff9dc6b4ca6c: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771
14:23:00.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.053085 28912 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 34a6cb78ecdf4d7a85465851c4c3bb04)
14:23:00.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.053573 28912 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:23:00.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.055655 28912 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:23:00.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:00.071112 28607 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:23:00.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.076104 28854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "bd010bbd952447cb8aab0a8e79691a65" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "b575acd823514305a1d3e6f006cb011b" is_pre_election: true
14:23:00.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.076889 28854 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd010bbd952447cb8aab0a8e79691a65 in term 9.
14:23:00.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.078235 28608 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b575acd823514305a1d3e6f006cb011b, bd010bbd952447cb8aab0a8e79691a65; no voters: 34a6cb78ecdf4d7a85465851c4c3bb04
14:23:00.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.079095 28912 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 10 FOLLOWER]: Leader pre-election won for term 11
14:23:00.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.079381 28912 raft_consensus.cc:493] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 34a6cb78ecdf4d7a85465851c4c3bb04)
14:23:00.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.079741 28912 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 10 FOLLOWER]: Advancing to term 11
14:23:00.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.084647 28912 raft_consensus.cc:515] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:23:00.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.086345 28912 leader_election.cc:290] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 11 election: Requested vote from peers 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771), b575acd823514305a1d3e6f006cb011b (127.25.218.129:33083)
14:23:00.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.087508 28854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "747213ca655547928c53ff9dc6b4ca6c" candidate_uuid: "bd010bbd952447cb8aab0a8e79691a65" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "b575acd823514305a1d3e6f006cb011b"
14:23:00.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.088196 28854 raft_consensus.cc:3060] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 9 FOLLOWER]: Advancing to term 11
14:23:00.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:00.090179 28607 leader_election.cc:336] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111)
14:23:00.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.094959 28854 raft_consensus.cc:2468] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd010bbd952447cb8aab0a8e79691a65 in term 11.
14:23:00.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.095988 28608 leader_election.cc:304] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b575acd823514305a1d3e6f006cb011b, bd010bbd952447cb8aab0a8e79691a65; no voters: 34a6cb78ecdf4d7a85465851c4c3bb04
14:23:00.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.096661 28912 raft_consensus.cc:2804] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 11 FOLLOWER]: Leader election won for term 11
14:23:00.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.097939 28912 raft_consensus.cc:697] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [term 11 LEADER]: Becoming Leader. State: Replica: bd010bbd952447cb8aab0a8e79691a65, State: Running, Role: LEADER
14:23:00.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.098727 28912 consensus_queue.cc:237] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:23:00.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.108080 26673 catalog_manager.cc:5697] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 reported cstate change: term changed from 10 to 11, leader changed from 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130) to bd010bbd952447cb8aab0a8e79691a65 (127.25.218.131). New cstate: current_term: 11 leader_uuid: "bd010bbd952447cb8aab0a8e79691a65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } health_report { overall_health: HEALTHY } } }
14:23:00.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.557915 28854 raft_consensus.cc:1275] T 747213ca655547928c53ff9dc6b4ca6c P b575acd823514305a1d3e6f006cb011b [term 11 FOLLOWER]: Refusing update from remote peer bd010bbd952447cb8aab0a8e79691a65: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
14:23:00.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.559544 28912 consensus_queue.cc:1048] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
14:23:00.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:00.576179 28931 mvcc.cc:204] Tried to move back new op lower bound from 7287251664144097280 to 7287251663688282112. Current Snapshot: MvccSnapshot[applied={T|T < 7287251664144097280}]
14:23:00.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:00.584779 28607 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 -> Peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Couldn't send request to peer 34a6cb78ecdf4d7a85465851c4c3bb04. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:23:01.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:01.151367 28607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111) [suppressed 3 similar messages]
14:23:02.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:02.922870 28607 consensus_peers.cc:597] T 747213ca655547928c53ff9dc6b4ca6c P bd010bbd952447cb8aab0a8e79691a65 -> Peer 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771): Couldn't send request to peer 34a6cb78ecdf4d7a85465851c4c3bb04. Status: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:23:03.153 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 747213ca655547928c53ff9dc6b4ca6c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513' with partition [<start>, <end>)
14:23:03.154 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=747213ca655547928c53ff9dc6b4ca6c@[34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771),b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083),bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)[L]]}]
14:23:03.154 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=747213ca655547928c53ff9dc6b4ca6c, attempt=14, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=4491), Traces: [0ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [3ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [3ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [11ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [12ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [14ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [14ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [30ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [31ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [32ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [32ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [70ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [71ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [73ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [73ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [190ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [191ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [193ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [193ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [406ms] refreshing cache from master, [406ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [411ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [412ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [414ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [414ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [586ms] refreshing cache from master, [586ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [590ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, [591ms] sending RPC to server 34a6cb78ecdf4d7a85465851c4c3bb04, [592ms] received response from server 34a6cb78ecdf4d7a85465851c4c3bb04: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [593ms] delaying RPC due to: Network error: Failed to connect to peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771): Connection refused: /127.25.218.130:39771, [4487ms] refreshing cache from master, [4487ms] Sub RPC GetTableLocations: sending RPC to server master-127.25.218.188:34713, [4491ms] Sub RPC GetTableLocations: received response from server master-127.25.218.188:34713: OK, deferred=Deferred@1976998391(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$228/0x00007fd7cc332738@7ce48c1f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434) -> (continuation of Deferred@1045002411 after retry RPC@385238341) -> (continuation of Deferred@2125812153 after retry RPC@1515056967) -> (continuation of Deferred@1464178828 after retry RPC@1648849652) -> (continuation of Deferred@205662099 after retry RPC@517478150) -> (continuation of Deferred@307706760 after retry RPC@1290366691), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@35018e6f -> wakeup thread Time-limited test -> (continuation of Deferred@1253109845 after retry RPC@1945857434) -> (continuation of Deferred@1045002411 after retry RPC@385238341) -> (continuation of Deferred@2125812153 after retry RPC@1515056967) -> (continuation of Deferred@1464178828 after retry RPC@1648849652) -> (continuation of Deferred@205662099 after retry RPC@517478150) -> (continuation of Deferred@307706760 after retry RPC@1290366691))) after lookup
14:23:03.155 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] connecting to peer
14:23:03.156 [DEBUG - kudu-nio-2] (Connection.java:757) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] Successfully connected to peer
14:23:03.178 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x4ffde289, L:/127.0.0.1:39836 - R:/127.25.218.131:44025] using TOKEN/null
14:23:03.241 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.25.218.130:39771
14:23:03.241 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:03.243052 26474 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/logs
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:23:03.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.130:39771
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.130
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37055
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.130
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:23:03.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:23:03.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:03.566308 28951 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:23:03.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:03.566871 28951 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:23:03.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:03.567519 28951 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:23:03.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:03.602739 28951 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:23:03.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:03.604020 28951 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.218.130
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:03.644812 28951 tablet_server_runner.cc:78] Tablet server non-default flags:
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.25.218.148:34335
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.25.218.130:39771
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.25.218.130
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37055
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.25.218.190:42765,127.25.218.189:41079,127.25.218.188:34713
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.25.218.130
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/logs
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 6bee9a2d051767e07c70d440ae169cb7177a10f8
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:23:03.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 14:03:54 UTC on bdcb31816ec0
14:23:03.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 15854
14:23:03.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:23:03.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:03.646080 28951 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:23:03.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:03.647780 28951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:23:03.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:03.661578 28959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:23:05.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:05.065444 28958 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 28951
14:23:05.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:05.153064 28958 kernel_stack_watchdog.cc:198] Thread 28951 stuck at /home/jenkins-slave/workspace/build_and_test@2/src/kudu/util/thread.cc:642 for 401ms:
14:23:05.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:23:05.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:23:05.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:23:05.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:23:05.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:03.662230 28960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:23:05.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:05.154351 28951 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.493s	user 0.000s	sys 0.003s
14:23:05.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:05.154799 28951 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.493s	user 0.000s	sys 0.003s
14:23:05.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:05.158005 28961 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1490 milliseconds
14:23:05.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:05.159514 28962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:23:05.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.159618 28951 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:23:05.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.161082 28951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:23:05.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.163285 28951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:23:05.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.164676 28951 hybrid_clock.cc:648] HybridClock initialized: now 1779114185164632 us; error 41 us; skew 500 ppm
14:23:05.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.165613 28951 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:23:05.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.173606 28951 webserver.cc:533] Webserver started at http://127.25.218.130:37055/ using document root <none> and password file <none>
14:23:05.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.174657 28951 fs_manager.cc:362] Metadata directory not provided
14:23:05.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.174926 28951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:23:05.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.183825 28951 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.004s	sys 0.003s
14:23:05.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.189594 28974 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:23:05.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.190918 28951 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.000s	sys 0.005s
14:23:05.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.191284 28951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data,/tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:23:05.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34a6cb78ecdf4d7a85465851c4c3bb04"
14:23:05.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:22:07 on dist-test-slave-45s3"
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.193374 28951 fs_report.cc:389] FS layout report
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/wal
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/data
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:23:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:23:05.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.261376 28951 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:23:05.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.263360 28951 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:23:05.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.263991 28951 kserver.cc:163] Server-wide thread pool size limit: 3276
14:23:05.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.266803 28951 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:23:05.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.272989 28981 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:23:05.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.282047 28951 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:23:05.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.282332 28951 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
14:23:05.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.282661 28951 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:23:05.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.289038 28951 ts_tablet_manager.cc:616] Registered 1 tablets
14:23:05.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.289310 28951 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.005s	sys 0.001s
14:23:05.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.289628 28981 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap starting.
14:23:05.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.377561 28981 log.cc:826] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Log is configured to *not* fsync() on all Append() calls
14:23:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.483568 28951 rpc_server.cc:307] RPC server started. Bound to: 127.25.218.130:39771
14:23:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.483844 29089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.218.130:39771 every 8 connection(s)
14:23:05.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.486392 28951 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskq0uqLp/test-tmp/mini-kudu-cluster15855824833936273380/ts-1/data/info.pb
14:23:05.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.488629 28981 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:23:05.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.489805 28981 tablet_bootstrap.cc:492] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Bootstrap complete.
14:23:05.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.491662 28981 ts_tablet_manager.cc:1403] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Time spent bootstrapping tablet: real 0.202s	user 0.136s	sys 0.060s
14:23:05.496 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:23:05.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.494796 26474 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu as pid 28951
14:23:05.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.519096 28981 raft_consensus.cc:359] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:23:05.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.522399 28981 raft_consensus.cc:740] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a6cb78ecdf4d7a85465851c4c3bb04, State: Initialized, Role: FOLLOWER
14:23:05.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.523321 28981 consensus_queue.cc:260] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a6cb78ecdf4d7a85465851c4c3bb04" member_type: VOTER last_known_addr { host: "127.25.218.130" port: 39771 } } peers { permanent_uuid: "b575acd823514305a1d3e6f006cb011b" member_type: VOTER last_known_addr { host: "127.25.218.129" port: 33083 } } peers { permanent_uuid: "bd010bbd952447cb8aab0a8e79691a65" member_type: VOTER last_known_addr { host: "127.25.218.131" port: 44025 } }
14:23:05.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.532810 28981 ts_tablet_manager.cc:1434] T 747213ca655547928c53ff9dc6b4ca6c P 34a6cb78ecdf4d7a85465851c4c3bb04: Time spent starting tablet: real 0.041s	user 0.037s	sys 0.000s
14:23:05.537 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:23:05.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.545243 29090 heartbeater.cc:344] Connected to a master server at 127.25.218.188:34713
14:23:05.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.545746 29090 heartbeater.cc:461] Registering TS with master...
14:23:05.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.547199 29090 heartbeater.cc:507] Master 127.25.218.188:34713 requested a full tablet report, sending...
14:23:05.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.548769 29091 heartbeater.cc:344] Connected to a master server at 127.25.218.189:41079
14:23:05.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.549144 29091 heartbeater.cc:461] Registering TS with master...
14:23:05.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.550124 29091 heartbeater.cc:507] Master 127.25.218.189:41079 requested a full tablet report, sending...
14:23:05.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.553017 29092 heartbeater.cc:344] Connected to a master server at 127.25.218.190:42765
14:23:05.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.553388 29092 heartbeater.cc:461] Registering TS with master...
14:23:05.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.554286 29092 heartbeater.cc:507] Master 127.25.218.190:42765 requested a full tablet report, sending...
14:23:05.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.556221 26594 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:23:05.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.561203 26674 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:23:05.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.564198 26518 ts_manager.cc:194] Re-registered known tserver with Master: 34a6cb78ecdf4d7a85465851c4c3bb04 (127.25.218.130:39771)
14:23:05.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.564651 26674 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.25.218.130:41739
14:23:05.566 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779114130513, tablet=747213ca655547928c53ff9dc6b4ca6c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:23:05.572 [DEBUG - main] (Connection.java:575) [peer master-127.25.218.190:42765(127.25.218.190:42765)] disconnecting while in state READY
14:23:05.573 [DEBUG - main] (Connection.java:575) [peer b575acd823514305a1d3e6f006cb011b(127.25.218.129:33083)] disconnecting while in state TERMINATED
14:23:05.573 [DEBUG - main] (Connection.java:575) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] disconnecting while in state READY
14:23:05.573 [DEBUG - main] (Connection.java:575) [peer 34a6cb78ecdf4d7a85465851c4c3bb04(127.25.218.130:39771)] disconnecting while in state TERMINATED
14:23:05.574 [DEBUG - main] (Connection.java:575) [peer master-127.25.218.189:41079(127.25.218.189:41079)] disconnecting while in state READY
14:23:05.574 [DEBUG - main] (Connection.java:575) [peer master-127.25.218.188:34713(127.25.218.188:34713)] disconnecting while in state READY
14:23:05.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.574620 29090 heartbeater.cc:499] Master 127.25.218.188:34713 was elected leader, sending a full tablet report...
14:23:05.580 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.25.218.189:41079(127.25.218.189:41079)] handling channelInactive
14:23:05.581 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.25.218.189:41079(127.25.218.189:41079)] cleaning up while in state READY due to: connection closed
14:23:05.581 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.25.218.188:34713(127.25.218.188:34713)] handling channelInactive
14:23:05.581 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.25.218.188:34713(127.25.218.188:34713)] cleaning up while in state READY due to: connection closed
14:23:05.583 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2704) Releasing all remaining resources
14:23:05.587 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.25.218.190:42765(127.25.218.190:42765)] handling channelInactive
14:23:05.587 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.25.218.190:42765(127.25.218.190:42765)] cleaning up while in state READY due to: connection closed
14:23:05.587 [DEBUG - kudu-nio-2] (Connection.java:239) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] handling channelInactive
14:23:05.587 [DEBUG - kudu-nio-2] (Connection.java:709) [peer bd010bbd952447cb8aab0a8e79691a65(127.25.218.131:44025)] cleaning up while in state READY due to: connection closed
14:23:05.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.588812 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 28763
14:23:05.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.616585 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 28951
14:23:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:05.655483 28969 negotiation.cc:336] Failed RPC negotiation. Trace:
14:23:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:23:05.645597 (+     0us) reactor.cc:678] Submitting negotiation task for client connection to 127.25.218.130:39771 (local address 127.25.218.131:50465)
14:23:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:23:05.646565 (+   968us) negotiation.cc:107] Waiting for socket to connect
14:23:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:23:05.646613 (+    48us) client_negotiation.cc:175] Beginning negotiation
14:23:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:23:05.646952 (+   339us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
14:23:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:23:05.654145 (+  7193us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.218.130:39771: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
14:23:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"client-negotiator.queue_time_us":333}
14:23:05.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.660540 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 28584
14:23:05.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:05.692252 26503 connection.cc:582] server connection from 127.25.218.131:50827 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
14:23:05.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.692734 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 26486
14:23:05.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.724495 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 26556
14:23:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:23:05.757136 26658 connection.cc:582] client connection to 127.25.218.189:41079 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
14:23:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:23:05.757521 26474 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq0uqLp/build/tsan/bin/kudu with pid 26631
14:23:06.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T14:23:06Z chronyd exiting

Time: 74.083

OK (1 test)

