JUnit version 4.13.2
.14:05:40.583 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1783519540575
14:05:40.638 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
14:05:40.700 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
14:05:40.712 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskB9LWx4/test-tmp
14:05:40.717 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
14:05:40.720 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskB9LWx4/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
14:05:40.915 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

14:05:41.093 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:05:41.094 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

14:05:41.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:41.100050 30075 mini_oidc.cc:150] Starting JWKS server
14:05:41.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:41.104206 30075 webserver.cc:533] Webserver started at http://127.0.0.1:43451/ using document root <none> and password file <none>
14:05:41.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:41.104811 30075 mini_oidc.cc:186] Starting OIDC Discovery server
14:05:41.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:41.107048 30075 webserver.cc:533] Webserver started at http://127.0.0.1:40341/ using document root <none> and password file <none>
14:05:41.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T14:05:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
14:05:41.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T14:05:41Z Disabled control of system clock
14:05:41.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:41.165956 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:41.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:41.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/wal
14:05:41.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/data
14:05:41.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:05:41.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:05:41.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:05:41.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:41.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:41.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:41.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/logs
14:05:41.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/data/info.pb
14:05:41.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:41.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:05:41.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:05:41.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:05:41.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:05:41.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:41.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:41.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:41.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:05:41.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:05:41.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:05:41.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:05:41.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.254:40897
14:05:41.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.254
14:05:41.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:41.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:41.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:41.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:41.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:41.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:05:41.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:41.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:05:41.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43451/jwks.json with env {}
14:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:41.456594 30088 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:41.457124 30088 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:41.457563 30088 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:05:41.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:41.489264 30088 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:05:41.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:41.489562 30088 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:05:41.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:41.489756 30088 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:05:41.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:41.489997 30088 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:05:41.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:41.490216 30088 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:05:41.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:41.490458 30088 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:43451/jwks.json
14:05:41.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:41.526598 30088 master_runner.cc:387] Master server non-default flags:
14:05:41.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:41.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/data
14:05:41.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/wal
14:05:41.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:05:41.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:41.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:41.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:41.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:05:41.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.254:40897
14:05:41.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:05:41.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:05:41.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:05:41.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43451/jwks.json
14:05:41.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:41.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:41.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/data/info.pb
14:05:41.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.254
14:05:41.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:41.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:05:41.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:41.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:05:41.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:05:41.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:41.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/logs
14:05:41.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:41.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:05:41.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:05:41.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:05:41.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:05:41.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:05:41.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:05:41.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:05:41.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:05:41.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:05:41.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:41.527907 30088 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:05:41.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:41.529572 30088 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:05:41.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:41.541998 30094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:42.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:41.543467 30095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:42.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:42.657485 30096 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1112 milliseconds
14:05:42.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:42.659137 30097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:42.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:42.662611 30088 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.121s	user 0.412s	sys 0.708s
14:05:42.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:42.662945 30088 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.121s	user 0.412s	sys 0.708s
14:05:42.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.663193 30088 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:05:42.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.664393 30088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:05:42.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.667379 30088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:05:42.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.668850 30088 hybrid_clock.cc:648] HybridClock initialized: now 1783519542668761 us; error 85 us; skew 500 ppm
14:05:42.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.669939 30088 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:05:42.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.678324 30088 webserver.cc:533] Webserver started at http://127.29.94.254:36969/ using document root <none> and password file <none>
14:05:42.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.679355 30088 fs_manager.cc:362] Metadata directory not provided
14:05:42.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.679560 30088 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:05:42.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.680164 30088 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:05:42.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.684719 30088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/data/instance:
14:05:42.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d747074cfc734dd2b5df7fece3198f95"
14:05:42.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:42 on dist-test-slave-59nb"
14:05:42.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.686033 30088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/wal/instance:
14:05:42.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d747074cfc734dd2b5df7fece3198f95"
14:05:42.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:42 on dist-test-slave-59nb"
14:05:42.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.696843 30088 fs_manager.cc:696] Time spent creating directory manager: real 0.010s	user 0.004s	sys 0.004s
14:05:42.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.703990 30104 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:05:42.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.705374 30088 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.004s	sys 0.000s
14:05:42.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.705754 30088 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/wal
14:05:42.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d747074cfc734dd2b5df7fece3198f95"
14:05:42.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:42 on dist-test-slave-59nb"
14:05:42.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.706225 30088 fs_report.cc:389] FS layout report
14:05:42.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:05:42.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/wal
14:05:42.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/wal
14:05:42.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/data/data
14:05:42.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:05:42.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:05:42.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:05:42.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:05:42.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:05:42.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:05:42.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:05:42.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:05:42.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:05:42.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:05:42.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:05:42.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.785187 30088 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:05:42.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.787125 30088 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:05:42.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.787647 30088 kserver.cc:163] Server-wide thread pool size limit: 3276
14:05:42.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.864828 30088 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.254:40897
14:05:42.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.864938 30155 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.254:40897 every 8 connection(s)
14:05:42.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.868224 30088 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/data/info.pb
14:05:42.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.873164 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 30088
14:05:42.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.873679 30075 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-0/wal/instance
14:05:42.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.875385 30156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:05:42.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.878605 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:42.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:42.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/wal
14:05:42.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/data
14:05:42.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:05:42.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:05:42.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:05:42.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:42.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:42.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:42.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/logs
14:05:42.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/data/info.pb
14:05:42.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:42.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:05:42.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:05:42.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:05:42.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:05:42.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:42.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:42.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:42.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:05:42.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:05:42.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:05:42.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:05:42.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.253:44767
14:05:42.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.253
14:05:42.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:42.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:42.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:42.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:42.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:42.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:05:42.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:42.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:05:42.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43451/jwks.json with env {}
14:05:42.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.888644 30156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } has no permanent_uuid. Determining permanent_uuid...
14:05:42.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.926187 30156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } has no permanent_uuid. Determining permanent_uuid...
14:05:42.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:42.928731 30108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.253:44767: connect: Connection refused (error 111)
14:05:42.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:42.932926 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.253:44767: Network error: Client connection negotiation failed: client connection to 127.29.94.253:44767: connect: Connection refused (error 111)
14:05:42.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:42.982586 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } attempt: 1
14:05:42.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:42.988076 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.253:44767: Network error: Client connection negotiation failed: client connection to 127.29.94.253:44767: connect: Connection refused (error 111)
14:05:43.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:43.056521 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } attempt: 2
14:05:43.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.061609 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.253:44767: Network error: Client connection negotiation failed: client connection to 127.29.94.253:44767: connect: Connection refused (error 111)
14:05:43.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:43.153003 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } attempt: 3
14:05:43.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.156948 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.253:44767: Network error: Client connection negotiation failed: client connection to 127.29.94.253:44767: connect: Connection refused (error 111)
14:05:43.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.205818 30158 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:05:43.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.206394 30158 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:05:43.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.206909 30158 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:05:43.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.238431 30158 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:05:43.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.238705 30158 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:05:43.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.238969 30158 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:05:43.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.239173 30158 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:05:43.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.239353 30158 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:05:43.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.239500 30158 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:43451/jwks.json
14:05:43.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:43.280169 30158 master_runner.cc:387] Master server non-default flags:
14:05:43.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:43.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:43.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:43.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:43.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/data
14:05:43.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/wal
14:05:43.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:05:43.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:43.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:43.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:43.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:05:43.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.253:44767
14:05:43.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:05:43.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:05:43.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:05:43.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43451/jwks.json
14:05:43.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:43.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:43.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/data/info.pb
14:05:43.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.253
14:05:43.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:43.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:05:43.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:43.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:05:43.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:05:43.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:43.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/logs
14:05:43.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:43.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:05:43.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:05:43.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:05:43.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:05:43.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:05:43.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:05:43.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:05:43.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:05:43.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:05:43.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:43.281590 30158 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:05:43.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:43.283236 30158 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:05:43.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.297179 30167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:43.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:43.300653 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } attempt: 4
14:05:43.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.308655 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.253:44767: Network error: Client connection negotiation failed: client connection to 127.29.94.253:44767: connect: Connection refused (error 111)
14:05:43.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:43.608114 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } attempt: 5
14:05:43.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.617074 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.253:44767: Network error: Client connection negotiation failed: client connection to 127.29.94.253:44767: connect: Connection refused (error 111)
14:05:44.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.164599 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } attempt: 6
14:05:44.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:44.189158 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.253:44767: Network error: Client connection negotiation failed: client connection to 127.29.94.253:44767: connect: Connection refused (error 111)
14:05:44.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:44.699725 30166 debug-util.cc:398] Leaking SignalData structure 0x7b0800028020 after lost signal to thread 30158
14:05:44.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:44.818205 30158 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.521s	user 0.545s	sys 0.970s
14:05:44.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:44.818622 30158 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.521s	user 0.545s	sys 0.970s
14:05:44.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:43.297848 30168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:44.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:44.820493 30170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:44.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:44.826444 30169 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1528 milliseconds
14:05:44.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.826529 30158 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:05:44.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.828178 30158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:05:44.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.831308 30158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:05:44.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.833146 30158 hybrid_clock.cc:648] HybridClock initialized: now 1783519544833077 us; error 78 us; skew 500 ppm
14:05:44.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.834007 30158 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:05:44.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.840258 30158 webserver.cc:533] Webserver started at http://127.29.94.253:33253/ using document root <none> and password file <none>
14:05:44.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.841238 30158 fs_manager.cc:362] Metadata directory not provided
14:05:44.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.841449 30158 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:05:44.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.842062 30158 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:05:44.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.846230 30158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/data/instance:
14:05:44.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6abacea8ea24415d8dd066ce3d987732"
14:05:44.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:44 on dist-test-slave-59nb"
14:05:44.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.847505 30158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/wal/instance:
14:05:44.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6abacea8ea24415d8dd066ce3d987732"
14:05:44.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:44 on dist-test-slave-59nb"
14:05:44.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.855351 30158 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.005s	sys 0.001s
14:05:44.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.862076 30179 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:05:44.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.863515 30158 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.000s
14:05:44.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.864223 30158 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/wal
14:05:44.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6abacea8ea24415d8dd066ce3d987732"
14:05:44.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:44 on dist-test-slave-59nb"
14:05:44.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.865314 30158 fs_report.cc:389] FS layout report
14:05:44.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:05:44.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/wal
14:05:44.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/wal
14:05:44.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/data/data
14:05:44.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:05:44.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:05:44.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:05:44.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:05:44.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:05:44.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:05:44.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:05:44.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:05:44.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:05:44.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:05:44.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:05:44.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.917305 30158 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:05:44.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.919310 30158 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:05:44.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.919905 30158 kserver.cc:163] Server-wide thread pool size limit: 3276
14:05:44.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.998013 30158 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.253:44767
14:05:44.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:44.998127 30230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.253:44767 every 8 connection(s)
14:05:45.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.000824 30158 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/data/info.pb
14:05:45.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.003540 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 30158
14:05:45.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.004014 30075 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-1/wal/instance
14:05:45.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.007131 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:45.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:45.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/wal
14:05:45.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/data
14:05:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:05:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:05:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:05:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/logs
14:05:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/data/info.pb
14:05:45.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:45.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:05:45.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:05:45.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:05:45.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:05:45.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:45.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:05:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:05:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:05:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:05:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.252:36353
14:05:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.252
14:05:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:05:45.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:45.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:05:45.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43451/jwks.json with env {}
14:05:45.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.008817 30231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:05:45.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.022934 30231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } has no permanent_uuid. Determining permanent_uuid...
14:05:45.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.046200 30231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } has no permanent_uuid. Determining permanent_uuid...
14:05:45.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.062017 30231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } has no permanent_uuid. Determining permanent_uuid...
14:05:45.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.064955 30180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.069690 30231 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.119232 30231 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 1
14:05:45.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.124418 30231 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.192941 30231 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 2
14:05:45.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.197464 30231 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.249716 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } attempt: 7
14:05:45.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.263703 30156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } has no permanent_uuid. Determining permanent_uuid...
14:05:45.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.269910 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.288959 30231 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 3
14:05:45.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.294529 30231 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.328363 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 1
14:05:45.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.333371 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.335196 30233 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:05:45.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.335696 30233 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:05:45.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.336325 30233 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:05:45.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.368770 30233 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:05:45.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.369050 30233 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:05:45.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.369254 30233 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:05:45.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.369441 30233 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:05:45.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.369619 30233 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:05:45.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.369755 30233 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:43451/jwks.json
14:05:45.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.406041 30233 master_runner.cc:387] Master server non-default flags:
14:05:45.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:45.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:45.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:45.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:45.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/data
14:05:45.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/wal
14:05:45.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:05:45.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:45.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:45.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:45.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:05:45.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.252:36353
14:05:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:05:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:05:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:05:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43451/jwks.json
14:05:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/data/info.pb
14:05:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.252
14:05:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/logs
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:05:45.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:05:45.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.407445 30233 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:05:45.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.409538 30233 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:05:45.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.414882 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 2
14:05:45.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.420486 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.423161 30245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:45.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.438098 30231 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 4
14:05:45.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.446358 30231 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.505957 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 3
14:05:45.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.515671 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.656195 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 4
14:05:45.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.664166 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.746750 30231 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 5
14:05:45.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.753549 30231 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:45.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:45.958700 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 5
14:05:45.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.972075 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:46.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.301090 30231 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 6
14:05:46.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:46.339608 30231 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:46.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.524586 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 6
14:05:46.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:46.561743 30156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.94.252:36353: Network error: Client connection negotiation failed: client connection to 127.29.94.252:36353: connect: Connection refused (error 111)
14:05:46.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:45.424726 30246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:46.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:46.759415 30248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:46.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:46.763290 30233 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.340s	user 0.496s	sys 0.834s
14:05:46.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:46.763537 30247 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1332 milliseconds
14:05:46.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:46.763737 30233 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.340s	user 0.496s	sys 0.834s
14:05:46.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.764075 30233 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:05:46.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.765429 30233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:05:46.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.768345 30233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:05:46.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.769837 30233 hybrid_clock.cc:648] HybridClock initialized: now 1783519546769783 us; error 62 us; skew 500 ppm
14:05:46.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.770788 30233 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:05:46.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.780437 30233 webserver.cc:533] Webserver started at http://127.29.94.252:35331/ using document root <none> and password file <none>
14:05:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.781620 30233 fs_manager.cc:362] Metadata directory not provided
14:05:46.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.781860 30233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:05:46.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.782442 30233 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:05:46.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.787109 30233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/data/instance:
14:05:46.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "891ca261d85e495595958d4876528f6c"
14:05:46.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:46 on dist-test-slave-59nb"
14:05:46.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.788580 30233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/wal/instance:
14:05:46.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "891ca261d85e495595958d4876528f6c"
14:05:46.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:46 on dist-test-slave-59nb"
14:05:46.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.797986 30233 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.007s	sys 0.000s
14:05:46.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.805513 30259 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:05:46.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.807000 30233 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.002s	sys 0.004s
14:05:46.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.807485 30233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/wal
14:05:46.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "891ca261d85e495595958d4876528f6c"
14:05:46.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:46 on dist-test-slave-59nb"
14:05:46.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.807961 30233 fs_report.cc:389] FS layout report
14:05:46.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:05:46.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/wal
14:05:46.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/wal
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/data/data
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:05:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:05:46.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.893847 30233 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:05:46.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.895354 30233 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:05:46.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.895741 30233 kserver.cc:163] Server-wide thread pool size limit: 3276
14:05:46.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.975870 30233 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.252:36353
14:05:46.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.975988 30310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.252:36353 every 8 connection(s)
14:05:46.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.978451 30233 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/data/info.pb
14:05:46.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.985567 30311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:05:46.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.988067 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 30233
14:05:46.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:46.988423 30075 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/master-2/wal/instance
14:05:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.001726 30311 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } has no permanent_uuid. Determining permanent_uuid...
14:05:47.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.036353 30311 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } has no permanent_uuid. Determining permanent_uuid...
14:05:47.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.046844 30311 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } has no permanent_uuid. Determining permanent_uuid...
14:05:47.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.064280 30311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c: Bootstrap starting.
14:05:47.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.070653 30311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c: Neither blocks nor log segments found. Creating new log.
14:05:47.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.072274 30311 log.cc:826] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c: Log is configured to *not* fsync() on all Append() calls
14:05:47.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.077158 30311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c: No bootstrap required, opened a new log
14:05:47.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.095623 30311 raft_consensus.cc:359] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } }
14:05:47.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.096231 30311 raft_consensus.cc:385] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:05:47.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.096488 30311 raft_consensus.cc:740] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891ca261d85e495595958d4876528f6c, State: Initialized, Role: FOLLOWER
14:05:47.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.097369 30311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } }
14:05:47.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.100756 30320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } } }
14:05:47.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.101408 30320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [sys.catalog]: This master's current role is: FOLLOWER
14:05:47.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.102465 30311 sys_catalog.cc:565] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [sys.catalog]: configured and running, proceeding with master startup.
14:05:47.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:47.131044 30331 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:05:47.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:47.131428 30331 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:05:47.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.124912 30233 master_runner.cc:187] Error getting master registration for 127.29.94.254:40897: OK, instance_id { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" instance_seqno: 1783519542844204 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:05:47.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.149675 30233 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:05:47.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:47.151563 30265 tablet.cc:2406] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:05:47.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.400152 30231 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 7
14:05:47.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.435702 30231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732: Bootstrap starting.
14:05:47.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.447872 30231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732: Neither blocks nor log segments found. Creating new log.
14:05:47.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.451398 30231 log.cc:826] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732: Log is configured to *not* fsync() on all Append() calls
14:05:47.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.458266 30231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732: No bootstrap required, opened a new log
14:05:47.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.487594 30231 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } }
14:05:47.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.488478 30231 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:05:47.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.488768 30231 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6abacea8ea24415d8dd066ce3d987732, State: Initialized, Role: FOLLOWER
14:05:47.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.489826 30231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } }
14:05:47.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.492477 30335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } } }
14:05:47.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.493235 30335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [sys.catalog]: This master's current role is: FOLLOWER
14:05:47.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.494328 30231 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [sys.catalog]: configured and running, proceeding with master startup.
14:05:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:47.528115 30346 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:05:47.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:47.529220 30346 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:05:47.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.529194 30335 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:05:47.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.530058 30335 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } }
14:05:47.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.532514 30320 raft_consensus.cc:493] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:05:47.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.533073 30320 raft_consensus.cc:515] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } }
14:05:47.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.538967 30206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "891ca261d85e495595958d4876528f6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6abacea8ea24415d8dd066ce3d987732" is_pre_election: true
14:05:47.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.540670 30320 leader_election.cc:290] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d747074cfc734dd2b5df7fece3198f95 (127.29.94.254:40897), 6abacea8ea24415d8dd066ce3d987732 (127.29.94.253:44767)
14:05:47.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.539769 30206 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 891ca261d85e495595958d4876528f6c in term 0.
14:05:47.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.552356 30263 leader_election.cc:304] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6abacea8ea24415d8dd066ce3d987732, 891ca261d85e495595958d4876528f6c; no voters: 
14:05:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.553942 30320 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 0 FOLLOWER]: Leader pre-election won for term 1
14:05:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.554641 30320 raft_consensus.cc:493] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:05:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.555330 30320 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 0 FOLLOWER]: Advancing to term 1
14:05:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.556425 30335 leader_election.cc:290] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d747074cfc734dd2b5df7fece3198f95 (127.29.94.254:40897), 891ca261d85e495595958d4876528f6c (127.29.94.252:36353)
14:05:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.558285 30286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6abacea8ea24415d8dd066ce3d987732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891ca261d85e495595958d4876528f6c" is_pre_election: true
14:05:47.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.582266 30320 raft_consensus.cc:515] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } }
14:05:47.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.584336 30286 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6abacea8ea24415d8dd066ce3d987732 in current term 1: Already voted for candidate 891ca261d85e495595958d4876528f6c in this term.
14:05:47.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.585377 30320 leader_election.cc:290] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [CANDIDATE]: Term 1 election: Requested vote from peers d747074cfc734dd2b5df7fece3198f95 (127.29.94.254:40897), 6abacea8ea24415d8dd066ce3d987732 (127.29.94.253:44767)
14:05:47.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.590492 30206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "891ca261d85e495595958d4876528f6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6abacea8ea24415d8dd066ce3d987732"
14:05:47.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.523207 30158 master_runner.cc:187] Error getting master registration for 127.29.94.254:40897: OK, instance_id { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" instance_seqno: 1783519542844204 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:05:47.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.591197 30206 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [term 0 FOLLOWER]: Advancing to term 1
14:05:47.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.591694 30158 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:05:47.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.595193 30156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } attempt: 7
14:05:47.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.599771 30206 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 891ca261d85e495595958d4876528f6c in term 1.
14:05:47.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.601711 30263 leader_election.cc:304] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6abacea8ea24415d8dd066ce3d987732, 891ca261d85e495595958d4876528f6c; no voters: 
14:05:47.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.602602 30320 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 1 FOLLOWER]: Leader election won for term 1
14:05:47.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.610145 30320 raft_consensus.cc:697] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [term 1 LEADER]: Becoming Leader. State: Replica: 891ca261d85e495595958d4876528f6c, State: Running, Role: LEADER
14:05:47.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.612078 30320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } }
14:05:47.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.628473 30156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95: Bootstrap starting.
14:05:47.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.640368 30156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95: Neither blocks nor log segments found. Creating new log.
14:05:47.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.643767 30349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 891ca261d85e495595958d4876528f6c. Latest consensus state: current_term: 1 leader_uuid: "891ca261d85e495595958d4876528f6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } } }
14:05:47.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.644559 30349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [sys.catalog]: This master's current role is: LEADER
14:05:47.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.651780 30156 log.cc:826] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95: Log is configured to *not* fsync() on all Append() calls
14:05:47.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.662855 30351 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
14:05:47.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.673941 30156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95: No bootstrap required, opened a new log
14:05:47.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:47.686389 30185 tablet.cc:2406] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:05:47.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.688647 30351 catalog_manager.cc:1520] Initializing Kudu cluster ID...
14:05:47.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.715456 30206 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [term 1 FOLLOWER]: Refusing update from remote peer 891ca261d85e495595958d4876528f6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:05:47.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.718134 30349 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
14:05:47.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.766018 30335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 891ca261d85e495595958d4876528f6c. Latest consensus state: current_term: 1 leader_uuid: "891ca261d85e495595958d4876528f6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } } }
14:05:47.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.767506 30335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [sys.catalog]: This master's current role is: FOLLOWER
14:05:47.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.767278 30156 raft_consensus.cc:359] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } }
14:05:47.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.775885 30156 raft_consensus.cc:385] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:05:47.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.779459 30156 raft_consensus.cc:740] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d747074cfc734dd2b5df7fece3198f95, State: Initialized, Role: FOLLOWER
14:05:47.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.783818 30349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "891ca261d85e495595958d4876528f6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } } }
14:05:47.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.784560 30349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [sys.catalog]: This master's current role is: LEADER
14:05:47.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.787297 30156 consensus_queue.cc:260] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } }
14:05:47.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.791296 30351 catalog_manager.cc:1383] Generated new cluster ID: 564bdbc6976e46a8a0e2aa0a8b8acb4c
14:05:47.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.791620 30351 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
14:05:47.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.820880 30359 sys_catalog.cc:455] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } } }
14:05:47.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.823478 30359 sys_catalog.cc:458] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [sys.catalog]: This master's current role is: FOLLOWER
14:05:47.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.823936 30359 raft_consensus.cc:493] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:05:47.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.824409 30359 raft_consensus.cc:515] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } }
14:05:47.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.832958 30206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d747074cfc734dd2b5df7fece3198f95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6abacea8ea24415d8dd066ce3d987732" is_pre_election: true
14:05:47.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.835639 30335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "891ca261d85e495595958d4876528f6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } } }
14:05:47.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.836369 30335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [sys.catalog]: This master's current role is: FOLLOWER
14:05:47.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.841204 30286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d747074cfc734dd2b5df7fece3198f95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891ca261d85e495595958d4876528f6c" is_pre_election: true
14:05:47.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.841878 30156 sys_catalog.cc:565] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [sys.catalog]: configured and running, proceeding with master startup.
14:05:47.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.842057 30129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "891ca261d85e495595958d4876528f6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d747074cfc734dd2b5df7fece3198f95"
14:05:47.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.842882 30129 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [term 0 FOLLOWER]: Advancing to term 1
14:05:47.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.840408 30359 leader_election.cc:290] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6abacea8ea24415d8dd066ce3d987732 (127.29.94.253:44767), 891ca261d85e495595958d4876528f6c (127.29.94.252:36353)
14:05:47.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.851125 30130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "891ca261d85e495595958d4876528f6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d747074cfc734dd2b5df7fece3198f95" is_pre_election: true
14:05:47.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.853173 30105 leader_election.cc:304] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d747074cfc734dd2b5df7fece3198f95; no voters: 6abacea8ea24415d8dd066ce3d987732, 891ca261d85e495595958d4876528f6c
14:05:47.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.847088 30131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6abacea8ea24415d8dd066ce3d987732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d747074cfc734dd2b5df7fece3198f95" is_pre_election: true
14:05:47.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.866775 30129 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 891ca261d85e495595958d4876528f6c in term 1.
14:05:47.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.868744 30182 leader_election.cc:304] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6abacea8ea24415d8dd066ce3d987732; no voters: 891ca261d85e495595958d4876528f6c, d747074cfc734dd2b5df7fece3198f95
14:05:47.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.869474 30359 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
14:05:47.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.870497 30335 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
14:05:47.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.871879 30128 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [term 1 FOLLOWER]: Refusing update from remote peer 891ca261d85e495595958d4876528f6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:05:47.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.879498 30351 catalog_manager.cc:1406] Generated new certificate authority record
14:05:47.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.882624 30351 catalog_manager.cc:1540] Loading token signing keys...
14:05:47.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.892833 30349 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:05:47.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.933413 30364 mvcc.cc:204] Tried to move back new op lower bound from 7305296067418566656 to 7305296067034681344. Current Snapshot: MvccSnapshot[applied={T|T < 7305296067418566656}]
14:05:47.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.936738 30351 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c: Generated new TSK 0
14:05:47.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.940302 30351 catalog_manager.cc:1550] Initializing in-progress tserver states...
14:05:47.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.959448 30357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "891ca261d85e495595958d4876528f6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } } }
14:05:47.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.959599 30359 sys_catalog.cc:455] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 891ca261d85e495595958d4876528f6c. Latest consensus state: current_term: 1 leader_uuid: "891ca261d85e495595958d4876528f6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } } }
14:05:47.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.960448 30357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [sys.catalog]: This master's current role is: FOLLOWER
14:05:47.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.960523 30359 sys_catalog.cc:458] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95 [sys.catalog]: This master's current role is: FOLLOWER
14:05:47.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.963346 30320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "891ca261d85e495595958d4876528f6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d747074cfc734dd2b5df7fece3198f95" member_type: VOTER last_known_addr { host: "127.29.94.254" port: 40897 } } peers { permanent_uuid: "6abacea8ea24415d8dd066ce3d987732" member_type: VOTER last_known_addr { host: "127.29.94.253" port: 44767 } } peers { permanent_uuid: "891ca261d85e495595958d4876528f6c" member_type: VOTER last_known_addr { host: "127.29.94.252" port: 36353 } } }
14:05:47.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.964123 30320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891ca261d85e495595958d4876528f6c [sys.catalog]: This master's current role is: LEADER
14:05:47.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:47.981945 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:47.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:47.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:05:47.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data
14:05:47.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/logs
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:05:47.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.193:0
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.193
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.193
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:05:48.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:48.092541 30375 catalog_manager.cc:1295] Loaded cluster ID: 564bdbc6976e46a8a0e2aa0a8b8acb4c
14:05:48.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:48.092850 30375 catalog_manager.cc:1588] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95: loading cluster ID for follower catalog manager: success
14:05:48.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:48.101502 30375 catalog_manager.cc:1610] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95: acquiring CA information for follower catalog manager: success
14:05:48.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:48.107931 30375 catalog_manager.cc:1638] T 00000000000000000000000000000000 P d747074cfc734dd2b5df7fece3198f95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:05:48.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:48.556622 30346 catalog_manager.cc:1295] Loaded cluster ID: 564bdbc6976e46a8a0e2aa0a8b8acb4c
14:05:48.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:48.557395 30346 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732: loading cluster ID for follower catalog manager: success
14:05:48.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:48.565497 30346 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732: acquiring CA information for follower catalog manager: success
14:05:48.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:48.578675 30346 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 6abacea8ea24415d8dd066ce3d987732: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:05:48.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:48.710657 30374 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:05:48.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:48.712332 30374 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:05:48.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:48.713532 30374 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:05:48.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:48.780848 30374 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:05:48.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:48.783138 30374 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.193
14:05:48.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:48.856191 30374 tablet_server_runner.cc:78] Tablet server non-default flags:
14:05:48.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:48.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:48.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:48.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:48.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data
14:05:48.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:05:48.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:48.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:48.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.193:0
14:05:48.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:05:48.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:05:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.193
14:05:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:05:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:05:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:05:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.193
14:05:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/logs
14:05:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:05:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:05:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:05:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:05:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:05:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:05:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:05:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:05:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:05:48.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:48.867599 30374 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:05:48.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:48.870955 30374 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:05:48.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:48.915753 30382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:51.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:48.938141 30383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:51.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:50.295425 30381 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 30374
14:05:51.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.005087 30381 kernel_stack_watchdog.cc:198] Thread 30374 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 358ms:
14:05:51.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:05:51.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:05:51.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:05:51.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:05:51.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.005725 30374 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 2.070s	user 0.002s	sys 0.001s
14:05:51.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.006101 30374 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 2.071s	user 0.002s	sys 0.001s
14:05:51.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.024461 30386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:51.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.025665 30374 server_base.cc:1061] running on GCE node
14:05:51.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.027350 30374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:05:51.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.033761 30374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:05:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.035343 30374 hybrid_clock.cc:648] HybridClock initialized: now 1783519551035256 us; error 110 us; skew 500 ppm
14:05:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.036428 30374 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:05:51.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.043991 30374 webserver.cc:533] Webserver started at http://127.29.94.193:35541/ using document root <none> and password file <none>
14:05:51.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.044931 30374 fs_manager.cc:362] Metadata directory not provided
14:05:51.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.045132 30374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:05:51.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.045538 30374 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:05:51.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.049995 30374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/instance:
14:05:51.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e1649b01489046e3a63ca563aad7869c"
14:05:51.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:51 on dist-test-slave-59nb"
14:05:51.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.051185 30374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal/instance:
14:05:51.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e1649b01489046e3a63ca563aad7869c"
14:05:51.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:51 on dist-test-slave-59nb"
14:05:51.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.059406 30374 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.002s	sys 0.005s
14:05:51.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.066108 30393 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:05:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.067245 30374 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.006s	sys 0.001s
14:05:51.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.067570 30374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:05:51.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e1649b01489046e3a63ca563aad7869c"
14:05:51.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:51 on dist-test-slave-59nb"
14:05:51.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.067982 30374 fs_report.cc:389] FS layout report
14:05:51.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:05:51.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/data
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:05:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:05:51.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.139477 30374 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:05:51.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.141165 30374 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:05:51.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.141600 30374 kserver.cc:163] Server-wide thread pool size limit: 3276
14:05:51.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.144765 30374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:05:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.149665 30374 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:05:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.149906 30374 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:05:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.150207 30374 ts_tablet_manager.cc:616] Registered 0 tablets
14:05:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.150470 30374 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:05:51.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.373322 30374 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.193:46741
14:05:51.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.373420 30505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.193:46741 every 8 connection(s)
14:05:51.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.376601 30374 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:05:51.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.379743 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 30374
14:05:51.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.380445 30075 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal/instance
14:05:51.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.401341 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/logs
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:05:51.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.194:0
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.194
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.194
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:05:51.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.455317 30507 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:05:51.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.455816 30507 heartbeater.cc:461] Registering TS with master...
14:05:51.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.456854 30507 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:05:51.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.460065 30196 ts_manager.cc:194] Registered new tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:05:51.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.461297 30506 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:05:51.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.461628 30506 heartbeater.cc:461] Registering TS with master...
14:05:51.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.462365 30506 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:05:51.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.465476 30276 ts_manager.cc:194] Registered new tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:05:51.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.467439 30276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.193:44219
14:05:51.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.476500 30508 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:05:51.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.476998 30508 heartbeater.cc:461] Registering TS with master...
14:05:51.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.477761 30508 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:05:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.482741 30121 ts_manager.cc:194] Registered new tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:05:51.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.777920 30516 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:05:51.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.778481 30516 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:05:51.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.778991 30516 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:05:51.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.813469 30516 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:05:51.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.814311 30516 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.194
14:05:51.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.852334 30516 tablet_server_runner.cc:78] Tablet server non-default flags:
14:05:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data
14:05:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:05:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:51.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.194:0
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.194
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.194
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/logs
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:05:51.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.853895 30516 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:05:51.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:51.855655 30516 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:05:51.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.870673 30523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:52.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:52.470409 30506 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:05:53.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.872927 30524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:53.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:51.875733 30526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:53.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:53.022553 30525 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1143 milliseconds
14:05:53.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.022653 30516 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:05:53.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.024595 30516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:05:53.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.027666 30516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:05:53.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.029111 30516 hybrid_clock.cc:648] HybridClock initialized: now 1783519553029054 us; error 82 us; skew 500 ppm
14:05:53.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.029939 30516 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:05:53.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.037681 30516 webserver.cc:533] Webserver started at http://127.29.94.194:33225/ using document root <none> and password file <none>
14:05:53.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.038686 30516 fs_manager.cc:362] Metadata directory not provided
14:05:53.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.038956 30516 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:05:53.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.039382 30516 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:05:53.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.044111 30516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/instance:
14:05:53.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "957569291a7c402594d391a83985d059"
14:05:53.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:53 on dist-test-slave-59nb"
14:05:53.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.045209 30516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal/instance:
14:05:53.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "957569291a7c402594d391a83985d059"
14:05:53.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:53 on dist-test-slave-59nb"
14:05:53.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.053431 30516 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.006s	sys 0.003s
14:05:53.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.059336 30533 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:05:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.060580 30516 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
14:05:53.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.060879 30516 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:05:53.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "957569291a7c402594d391a83985d059"
14:05:53.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:53 on dist-test-slave-59nb"
14:05:53.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.061223 30516 fs_report.cc:389] FS layout report
14:05:53.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:05:53.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:05:53.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:05:53.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/data
14:05:53.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:05:53.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:05:53.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:05:53.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:05:53.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:05:53.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:05:53.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:05:53.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:05:53.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:05:53.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:05:53.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:05:53.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.117641 30516 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:05:53.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.119642 30516 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:05:53.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.120051 30516 kserver.cc:163] Server-wide thread pool size limit: 3276
14:05:53.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.122620 30516 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:05:53.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.127506 30516 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:05:53.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.127699 30516 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:05:53.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.127892 30516 ts_tablet_manager.cc:616] Registered 0 tablets
14:05:53.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.128026 30516 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:05:53.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.279448 30516 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.194:43301
14:05:53.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.279604 30646 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.194:43301 every 8 connection(s)
14:05:53.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.281905 30516 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:05:53.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.287070 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 30516
14:05:53.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.287432 30075 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal/instance
14:05:53.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.298280 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:53.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/logs
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:05:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.195:0
14:05:53.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.195
14:05:53.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.195
14:05:53.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:53.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:53.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:53.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:53.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:53.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:05:53.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.343031 30649 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:05:53.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.343101 30648 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:05:53.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.343534 30647 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:05:53.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.343688 30649 heartbeater.cc:461] Registering TS with master...
14:05:53.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.343721 30648 heartbeater.cc:461] Registering TS with master...
14:05:53.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.343888 30647 heartbeater.cc:461] Registering TS with master...
14:05:53.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.345521 30647 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:05:53.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.345539 30649 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:05:53.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.345582 30648 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:05:53.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.348963 30276 ts_manager.cc:194] Registered new tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:05:53.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.349720 30121 ts_manager.cc:194] Registered new tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:05:53.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.350018 30196 ts_manager.cc:194] Registered new tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:05:53.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.350577 30276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.194:43573
14:05:53.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:53.639261 30657 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:05:53.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:53.639844 30657 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:05:53.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:53.640370 30657 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:05:53.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:53.673574 30657 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:05:53.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:53.674502 30657 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.195
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.711779 30657 tablet_server_runner.cc:78] Tablet server non-default flags:
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.195:0
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.195
14:05:53.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.195
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/logs
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:05:53.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.713506 30657 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:05:53.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:53.715234 30657 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:05:53.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:53.729518 30664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:54.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.354776 30647 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:05:54.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:53.731263 30663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:54.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:53.735059 30666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:54.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:54.841620 30665 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
14:05:54.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.841648 30657 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:05:54.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.846630 30657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:05:54.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.849332 30657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:05:54.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.850752 30657 hybrid_clock.cc:648] HybridClock initialized: now 1783519554850695 us; error 75 us; skew 500 ppm
14:05:54.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.851954 30657 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:05:54.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.858608 30657 webserver.cc:533] Webserver started at http://127.29.94.195:35565/ using document root <none> and password file <none>
14:05:54.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.859495 30657 fs_manager.cc:362] Metadata directory not provided
14:05:54.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.859696 30657 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:05:54.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.860134 30657 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:05:54.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.864277 30657 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/instance:
14:05:54.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:05:54.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:54 on dist-test-slave-59nb"
14:05:54.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.865417 30657 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal/instance:
14:05:54.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:05:54.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:54 on dist-test-slave-59nb"
14:05:54.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.872365 30657 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.006s
14:05:54.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.878360 30673 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:05:54.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.879839 30657 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.001s
14:05:54.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.880293 30657 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:05:54.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:05:54.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:54 on dist-test-slave-59nb"
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.880738 30657 fs_report.cc:389] FS layout report
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/data
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:05:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:05:54.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.945879 30657 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:05:54.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.947522 30657 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:05:54.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.947935 30657 kserver.cc:163] Server-wide thread pool size limit: 3276
14:05:54.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.951261 30657 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:05:54.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.955817 30657 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:05:54.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.956025 30657 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:05:54.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.956220 30657 ts_tablet_manager.cc:616] Registered 0 tablets
14:05:54.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:54.956353 30657 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:05:55.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.143282 30657 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.195:42127
14:05:55.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.143429 30785 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.195:42127 every 8 connection(s)
14:05:55.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.146057 30657 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:05:55.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.147210 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 30657
14:05:55.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.147722 30075 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal/instance
14:05:55.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.194206 30786 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:05:55.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.194790 30786 heartbeater.cc:461] Registering TS with master...
14:05:55.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.196357 30786 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:05:55.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.197786 30787 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:05:55.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.198141 30787 heartbeater.cc:461] Registering TS with master...
14:05:55.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.199646 30787 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:05:55.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.200352 30276 ts_manager.cc:194] Registered new tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:05:55.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.202132 30790 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:05:55.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.202147 30276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.195:40771
14:05:55.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.202445 30790 heartbeater.cc:461] Registering TS with master...
14:05:55.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.203851 30195 ts_manager.cc:194] Registered new tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:05:55.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.204025 30790 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:05:55.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.206647 30120 ts_manager.cc:194] Registered new tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:05:55.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:55.225744 30075 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
14:05:55.237 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:05:55.239 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

14:05:55.249 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.29.94.254"
      port: 40897
    }
    bound_http_address {
      host: "127.29.94.254"
      port: 36969
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.29.94.253"
      port: 44767
    }
    bound_http_address {
      host: "127.29.94.253"
      port: 33253
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.29.94.252"
      port: 36353
    }
    bound_http_address {
      host: "127.29.94.252"
      port: 35331
    }
  }
}

14:05:55.254 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

14:05:55.257 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.29.94.193"
      port: 46741
    }
    bound_http_address {
      host: "127.29.94.193"
      port: 35541
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.29.94.194"
      port: 43301
    }
    bound_http_address {
      host: "127.29.94.194"
      port: 33225
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.29.94.195"
      port: 42127
    }
    bound_http_address {
      host: "127.29.94.195"
      port: 35565
    }
  }
}

14:05:55.257 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
14:05:55.458 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
14:05:55.524 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
14:05:55.705 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.29.94.252:36353(127.29.94.252:36353)] connecting to peer
14:05:55.766 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.29.94.253:44767(127.29.94.253:44767)] connecting to peer
14:05:55.768 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.29.94.254:40897(127.29.94.254:40897)] connecting to peer
14:05:55.786 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.29.94.252:36353(127.29.94.252:36353)] Successfully connected to peer
14:05:55.791 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.29.94.254:40897(127.29.94.254:40897)] Successfully connected to peer
14:05:55.792 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.29.94.253:44767(127.29.94.253:44767)] Successfully connected to peer
14:05:55.889 [DEBUG - kudu-nio-1] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.29.94.253
14:05:55.891 [DEBUG - kudu-nio-2] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.29.94.254
14:05:55.893 [DEBUG - kudu-nio-0] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.29.94.252
14:05:56.172 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0x6502007f, L:/127.0.0.1:36782 - R:/127.29.94.254:40897] using SASL/PLAIN
14:05:56.172 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x398f9a90, L:/127.0.0.1:46100 - R:/127.29.94.252:36353] using SASL/PLAIN
14:05:56.172 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x1a68739c, L:/127.0.0.1:44368 - R:/127.29.94.253:44767] using SASL/PLAIN
14:05:56.206 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
14:05:56.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.206483 30786 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:05:56.208 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=552), Traces: [0ms] refreshing cache from master, [29ms] Sub RPC ConnectToMaster: sending RPC to server master-127.29.94.252:36353, [107ms] Sub RPC ConnectToMaster: sending RPC to server master-127.29.94.253:44767, [109ms] Sub RPC ConnectToMaster: sending RPC to server master-127.29.94.254:40897, [526ms] Sub RPC ConnectToMaster: received response from server master-127.29.94.254:40897: OK, [526ms] Sub RPC ConnectToMaster: received response from server master-127.29.94.253:44767: OK, [527ms] Sub RPC ConnectToMaster: received response from server master-127.29.94.252:36353: OK, deferred=null) after lookup
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.258309 30276 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46100:
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638"
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:05:56.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:05:56.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
14:05:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
14:05:56.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:05:56.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:05:56.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:56.262549 30276 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
14:05:56.346 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 01e264489a1643efb778ce4e4254ae1e
14:05:56.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.377908 30441 tablet_service.cc:1511] Processing CreateTablet for tablet 2a6d316bea6f4d1ba1862ba16772c63b (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638 [id=01e264489a1643efb778ce4e4254ae1e]), partition=RANGE (key) PARTITION UNBOUNDED
14:05:56.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.380112 30441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a6d316bea6f4d1ba1862ba16772c63b. 1 dirs total, 0 dirs full, 0 dirs failed
14:05:56.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.385411 30721 tablet_service.cc:1511] Processing CreateTablet for tablet 2a6d316bea6f4d1ba1862ba16772c63b (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638 [id=01e264489a1643efb778ce4e4254ae1e]), partition=RANGE (key) PARTITION UNBOUNDED
14:05:56.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.388115 30721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a6d316bea6f4d1ba1862ba16772c63b. 1 dirs total, 0 dirs full, 0 dirs failed
14:05:56.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.398391 30582 tablet_service.cc:1511] Processing CreateTablet for tablet 2a6d316bea6f4d1ba1862ba16772c63b (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638 [id=01e264489a1643efb778ce4e4254ae1e]), partition=RANGE (key) PARTITION UNBOUNDED
14:05:56.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.401216 30582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a6d316bea6f4d1ba1862ba16772c63b. 1 dirs total, 0 dirs full, 0 dirs failed
14:05:56.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.411568 30822 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap starting.
14:05:56.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.422467 30822 tablet_bootstrap.cc:654] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Neither blocks nor log segments found. Creating new log.
14:05:56.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.422582 30823 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Bootstrap starting.
14:05:56.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.425247 30822 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Log is configured to *not* fsync() on all Append() calls
14:05:56.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.428824 30824 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Bootstrap starting.
14:05:56.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.430467 30823 tablet_bootstrap.cc:654] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Neither blocks nor log segments found. Creating new log.
14:05:56.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.432757 30823 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Log is configured to *not* fsync() on all Append() calls
14:05:56.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.434600 30822 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: No bootstrap required, opened a new log
14:05:56.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.435142 30822 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Time spent bootstrapping tablet: real 0.024s	user 0.007s	sys 0.012s
14:05:56.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.435971 30824 tablet_bootstrap.cc:654] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Neither blocks nor log segments found. Creating new log.
14:05:56.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.438405 30824 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Log is configured to *not* fsync() on all Append() calls
14:05:56.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.446205 30823 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: No bootstrap required, opened a new log
14:05:56.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.446973 30823 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.004s
14:05:56.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.447708 30824 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: No bootstrap required, opened a new log
14:05:56.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.449546 30824 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Time spent bootstrapping tablet: real 0.021s	user 0.004s	sys 0.013s
14:05:56.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.464370 30822 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:56.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.465683 30822 raft_consensus.cc:385] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:05:56.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.466046 30822 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1649b01489046e3a63ca563aad7869c, State: Initialized, Role: FOLLOWER
14:05:56.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.466971 30822 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:56.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.470816 30822 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Time spent starting tablet: real 0.035s	user 0.024s	sys 0.008s
14:05:56.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.474181 30823 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:56.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.475167 30823 raft_consensus.cc:385] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:05:56.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.475507 30823 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c7b9c0f64b41c6b3f9a0e3084cf3e8, State: Initialized, Role: FOLLOWER
14:05:56.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.476532 30823 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:56.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.475966 30824 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:56.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.480414 30824 raft_consensus.cc:385] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:05:56.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.480949 30824 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957569291a7c402594d391a83985d059, State: Initialized, Role: FOLLOWER
14:05:56.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.482132 30824 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:56.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.494688 30823 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Time spent starting tablet: real 0.047s	user 0.029s	sys 0.013s
14:05:56.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.495914 30824 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Time spent starting tablet: real 0.046s	user 0.028s	sys 0.006s
14:05:56.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:56.580449 30652 tablet.cc:2406] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:05:56.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:56.673166 30509 tablet.cc:2406] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:05:56.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:56.693138 30793 tablet.cc:2406] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:05:56.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.788986 30828 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:05:56.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.789505 30828 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:56.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.791954 30828 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:05:56.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.800128 30830 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:05:56.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.801479 30830 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:56.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.805425 30830 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:05:56.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.806478 30741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" is_pre_election: true
14:05:56.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.807404 30741 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1649b01489046e3a63ca563aad7869c in term 0.
14:05:56.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.810626 30397 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1649b01489046e3a63ca563aad7869c, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: 
14:05:56.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.810546 30602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957569291a7c402594d391a83985d059" is_pre_election: true
14:05:56.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.811896 30828 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 0 FOLLOWER]: Leader pre-election won for term 1
14:05:56.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.812201 30828 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:05:56.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.812414 30828 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 0 FOLLOWER]: Advancing to term 1
14:05:56.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.814997 30602 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1649b01489046e3a63ca563aad7869c in term 0.
14:05:56.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.817703 30828 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:56.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.819555 30828 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 1 election: Requested vote from peers e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:05:56.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.821048 30602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957569291a7c402594d391a83985d059"
14:05:56.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.820284 30741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:05:56.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.821475 30602 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 0 FOLLOWER]: Advancing to term 1
14:05:56.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.821681 30741 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 0 FOLLOWER]: Advancing to term 1
14:05:56.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.828293 30602 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1649b01489046e3a63ca563aad7869c in term 1.
14:05:56.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.828927 30741 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1649b01489046e3a63ca563aad7869c in term 1.
14:05:56.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.830299 30397 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1649b01489046e3a63ca563aad7869c, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: 
14:05:56.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.831295 30828 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 1 FOLLOWER]: Leader election won for term 1
14:05:56.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.831672 30461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1649b01489046e3a63ca563aad7869c" is_pre_election: true
14:05:56.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.833217 30741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" is_pre_election: true
14:05:56.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.833904 30741 raft_consensus.cc:2393] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 957569291a7c402594d391a83985d059 in current term 1: Already voted for candidate e1649b01489046e3a63ca563aad7869c in this term.
14:05:56.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.833952 30828 raft_consensus.cc:697] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 1 LEADER]: Becoming Leader. State: Replica: e1649b01489046e3a63ca563aad7869c, State: Running, Role: LEADER
14:05:56.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.834923 30828 consensus_queue.cc:237] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:56.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.849377 30537 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 957569291a7c402594d391a83985d059; no voters: e1649b01489046e3a63ca563aad7869c, e2c7b9c0f64b41c6b3f9a0e3084cf3e8
14:05:56.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.850718 30830 raft_consensus.cc:2749] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
14:05:56.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:56.851296 30274 catalog_manager.cc:5697] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c reported cstate change: term changed from 0 to 1, leader changed from <none> to e1649b01489046e3a63ca563aad7869c (127.29.94.193). New cstate: current_term: 1 leader_uuid: "e1649b01489046e3a63ca563aad7869c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } health_report { overall_health: UNKNOWN } } }
14:05:56.918 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 01e264489a1643efb778ce4e4254ae1e
14:05:56.936 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:05:56.942 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299995, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:05:56.943 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=null) after lookup
14:05:56.953 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:05:56.955 [DEBUG - kudu-nio-3] (Connection.java:757) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] Successfully connected to peer
14:05:57.056 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0xde9459b0, L:/127.0.0.1:50682 - R:/127.29.94.193:46741] using TOKEN/null
14:05:57.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:57.073374 30602 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 1 FOLLOWER]: Refusing update from remote peer e1649b01489046e3a63ca563aad7869c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:05:57.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:57.073336 30741 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 1 FOLLOWER]: Refusing update from remote peer e1649b01489046e3a63ca563aad7869c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:05:57.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:57.074892 30835 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:05:57.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:57.075574 30828 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [LEADER]: Connected to new peer: Peer: permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:05:57.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:57.114300 30850 mvcc.cc:204] Tried to move back new op lower bound from 7305296105756352512 to 7305296104804380672. Current Snapshot: MvccSnapshot[applied={T|T < 7305296105756352512}]
14:05:57.286 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:05:57.297 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.94.193:46741
14:05:57.299 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:05:57.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:57.300086 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 30374
14:05:57.344 [DEBUG - kudu-nio-3] (Connection.java:239) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] handling channelInactive
14:05:57.345 [DEBUG - kudu-nio-3] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state READY due to: connection closed
14:05:57.358 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:05:57.360 [DEBUG - Time-limited test] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:05:57.363 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.363 [DEBUG - kudu-nio-4] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.364 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.382 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:05:57.382 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:05:57.383 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [5ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [5ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1374743450(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@22b7695e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46dca15d -> wakeup thread Time-limited test)) after lookup
14:05:57.384 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:05:57.386 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.387 [DEBUG - kudu-nio-5] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.387 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.402 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:05:57.402 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:05:57.403 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [5ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [5ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [24ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [28ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [28ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [42ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1374743450(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@22b7695e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46dca15d -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437))) after lookup
14:05:57.404 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:05:57.406 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.407 [DEBUG - kudu-nio-6] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.407 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.441 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:05:57.442 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:05:57.443 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [5ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [5ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [24ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [28ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [28ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [42ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [45ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [48ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [48ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [82ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1374743450(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@22b7695e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437) -> (continuation of Deferred@1393516654 after retry RPC@1385946472), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46dca15d -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437) -> (continuation of Deferred@1393516654 after retry RPC@1385946472))) after lookup
14:05:57.443 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:05:57.447 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.447 [DEBUG - kudu-nio-7] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.447 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.521 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:05:57.521 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:05:57.522 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=163), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [5ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [5ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [24ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [28ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [28ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [42ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [45ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [48ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [48ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [82ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [84ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [88ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [89ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [157ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [162ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1374743450(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@22b7695e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437) -> (continuation of Deferred@1393516654 after retry RPC@1385946472) -> (continuation of Deferred@969200021 after retry RPC@497347030), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46dca15d -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437) -> (continuation of Deferred@1393516654 after retry RPC@1385946472) -> (continuation of Deferred@969200021 after retry RPC@497347030))) after lookup
14:05:57.524 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:05:57.525 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.525 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:57.525 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:58.022 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:05:58.022 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:05:58.022 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=663), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [5ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [5ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [24ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [28ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [28ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [42ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [45ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [48ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [48ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [82ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [84ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [88ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [89ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [157ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [162ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [164ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [166ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [166ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [657ms] refreshing cache from master, [658ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [662ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1374743450(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@22b7695e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437) -> (continuation of Deferred@1393516654 after retry RPC@1385946472) -> (continuation of Deferred@969200021 after retry RPC@497347030) -> (continuation of Deferred@809615164 after retry RPC@332396450), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46dca15d -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437) -> (continuation of Deferred@1393516654 after retry RPC@1385946472) -> (continuation of Deferred@969200021 after retry RPC@497347030) -> (continuation of Deferred@809615164 after retry RPC@332396450))) after lookup
14:05:58.023 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:05:58.025 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:58.025 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:58.025 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:58.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.713827 30861 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1649b01489046e3a63ca563aad7869c)
14:05:58.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.714315 30861 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:58.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.716998 30861 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:05:58.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:58.720530 30677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:05:58.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.721886 30865 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1649b01489046e3a63ca563aad7869c)
14:05:58.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.722612 30865 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:58.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.724772 30865 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:05:58.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:58.725663 30677 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:05:58.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.728216 30741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" is_pre_election: true
14:05:58.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:58.728412 30537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:05:58.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.728828 30741 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957569291a7c402594d391a83985d059 in term 1.
14:05:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.730268 30602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "957569291a7c402594d391a83985d059" is_pre_election: true
14:05:58.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.730839 30602 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c7b9c0f64b41c6b3f9a0e3084cf3e8 in term 1.
14:05:58.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.731120 30537 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: 
14:05:58.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.731881 30865 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 1 FOLLOWER]: Leader pre-election won for term 2
14:05:58.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.731864 30677 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: e1649b01489046e3a63ca563aad7869c
14:05:58.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.732209 30865 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1649b01489046e3a63ca563aad7869c)
14:05:58.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.732527 30865 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 1 FOLLOWER]: Advancing to term 2
14:05:58.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.732946 30861 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 1 FOLLOWER]: Leader pre-election won for term 2
14:05:58.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.733297 30861 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1649b01489046e3a63ca563aad7869c)
14:05:58.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.733637 30861 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 1 FOLLOWER]: Advancing to term 2
14:05:58.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:58.733708 30537 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:05:58.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.737284 30865 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:58.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.738786 30865 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 2 election: Requested vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:05:58.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.738819 30861 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:05:58.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.740844 30861 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 2 election: Requested vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:05:58.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.742228 30602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "957569291a7c402594d391a83985d059"
14:05:58.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.742901 30602 raft_consensus.cc:2393] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e2c7b9c0f64b41c6b3f9a0e3084cf3e8 in current term 2: Already voted for candidate 957569291a7c402594d391a83985d059 in this term.
14:05:58.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.744402 30741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:05:58.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.745090 30741 raft_consensus.cc:2393] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 957569291a7c402594d391a83985d059 in current term 2: Already voted for candidate e2c7b9c0f64b41c6b3f9a0e3084cf3e8 in this term.
14:05:58.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:58.746762 30677 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:05:58.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.747219 30677 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: 957569291a7c402594d391a83985d059, e1649b01489046e3a63ca563aad7869c
14:05:58.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.748257 30861 raft_consensus.cc:2749] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
14:05:58.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:05:58.749820 30537 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:05:58.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.750161 30537 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 957569291a7c402594d391a83985d059; no voters: e1649b01489046e3a63ca563aad7869c, e2c7b9c0f64b41c6b3f9a0e3084cf3e8
14:05:58.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:05:58.750710 30865 raft_consensus.cc:2749] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
14:05:59.421 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:05:59.422 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:05:59.422 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2063), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [5ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [5ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [24ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [28ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [28ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [42ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [45ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [48ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [48ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [82ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [84ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [88ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [89ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [157ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [162ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [164ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [166ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [166ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [657ms] refreshing cache from master, [658ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [662ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [664ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [666ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [666ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2057ms] refreshing cache from master, [2057ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [2062ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1374743450(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@22b7695e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437) -> (continuation of Deferred@1393516654 after retry RPC@1385946472) -> (continuation of Deferred@969200021 after retry RPC@497347030) -> (continuation of Deferred@809615164 after retry RPC@332396450) -> (continuation of Deferred@971638549 after retry RPC@656361524), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46dca15d -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437) -> (continuation of Deferred@1393516654 after retry RPC@1385946472) -> (continuation of Deferred@969200021 after retry RPC@497347030) -> (continuation of Deferred@809615164 after retry RPC@332396450) -> (continuation of Deferred@971638549 after retry RPC@656361524))) after lookup
14:05:59.424 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:05:59.425 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:59.425 [DEBUG - kudu-nio-2] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:05:59.425 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.258936 30869 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:06:00.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.259342 30869 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:00.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.261297 30869 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:00.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.264154 30741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" is_pre_election: true
14:06:00.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.264650 30741 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957569291a7c402594d391a83985d059 in term 2.
14:06:00.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.265691 30537 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: 
14:06:00.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.266352 30869 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 2 FOLLOWER]: Leader pre-election won for term 3
14:06:00.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.266690 30869 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:06:00.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.267019 30869 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 2 FOLLOWER]: Advancing to term 3
14:06:00.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:00.267621 30537 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:06:00.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.271260 30869 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:00.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.272722 30869 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 3 election: Requested vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:00.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.274197 30741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:06:00.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.274636 30741 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 2 FOLLOWER]: Advancing to term 3
14:06:00.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:00.276090 30537 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:06:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.279325 30741 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957569291a7c402594d391a83985d059 in term 3.
14:06:00.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.280190 30537 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: e1649b01489046e3a63ca563aad7869c
14:06:00.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.280979 30869 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 3 FOLLOWER]: Leader election won for term 3
14:06:00.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.281996 30869 raft_consensus.cc:697] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 3 LEADER]: Becoming Leader. State: Replica: 957569291a7c402594d391a83985d059, State: Running, Role: LEADER
14:06:00.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.283048 30869 consensus_queue.cc:237] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:00.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.291147 30274 catalog_manager.cc:5697] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 reported cstate change: term changed from 1 to 3, leader changed from e1649b01489046e3a63ca563aad7869c (127.29.94.193) to 957569291a7c402594d391a83985d059 (127.29.94.194). New cstate: current_term: 3 leader_uuid: "957569291a7c402594d391a83985d059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } health_report { overall_health: HEALTHY } } }
14:06:00.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.676362 30741 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 3 FOLLOWER]: Refusing update from remote peer 957569291a7c402594d391a83985d059: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
14:06:00.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:00.677627 30869 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
14:06:00.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:00.693897 30537 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 -> Peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Couldn't send request to peer e1649b01489046e3a63ca563aad7869c. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:06:02.082 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:02.082 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:02.083 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=4723), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [5ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [5ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [24ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [28ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [28ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [42ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [45ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [48ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [48ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [82ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [84ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [88ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [89ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [157ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [162ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [164ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [166ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [166ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [657ms] refreshing cache from master, [658ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [662ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [664ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [666ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [666ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2057ms] refreshing cache from master, [2057ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [2062ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [2064ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2066ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2067ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [4717ms] refreshing cache from master, [4717ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [4722ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1374743450(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@22b7695e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437) -> (continuation of Deferred@1393516654 after retry RPC@1385946472) -> (continuation of Deferred@969200021 after retry RPC@497347030) -> (continuation of Deferred@809615164 after retry RPC@332396450) -> (continuation of Deferred@971638549 after retry RPC@656361524) -> (continuation of Deferred@1452902680 after retry RPC@186266903), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@46dca15d -> wakeup thread Time-limited test -> (continuation of Deferred@995468059 after retry RPC@444594437) -> (continuation of Deferred@1393516654 after retry RPC@1385946472) -> (continuation of Deferred@969200021 after retry RPC@497347030) -> (continuation of Deferred@809615164 after retry RPC@332396450) -> (continuation of Deferred@971638549 after retry RPC@656361524) -> (continuation of Deferred@1452902680 after retry RPC@186266903))) after lookup
14:06:02.085 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:02.086 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] Successfully connected to peer
14:06:02.113 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0xedf9ec7c, L:/127.0.0.1:36476 - R:/127.29.94.194:43301] using TOKEN/null
14:06:02.166 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.94.193:46741
14:06:02.168 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:06:02.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:02.170418 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:02.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:02.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:02.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data
14:06:02.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:06:02.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:06:02.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:06:02.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/logs
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.193:46741
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.193
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35541
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.193
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:02.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:06:02.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:02.470638 30905 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:06:02.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:02.471194 30905 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:06:02.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:02.471786 30905 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:06:02.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:02.504436 30905 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:06:02.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:02.505412 30905 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.193
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:02.542524 30905 tablet_server_runner.cc:78] Tablet server non-default flags:
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.193:46741
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.193
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35541
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:02.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.193
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/logs
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:06:02.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:02.544186 30905 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:06:02.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:02.545804 30905 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:06:02.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:02.557880 30911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:03.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:03.206312 30537 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 -> Peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Couldn't send request to peer e1649b01489046e3a63ca563aad7869c. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:06:03.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:02.559374 30912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:03.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:03.726029 30914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:03.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:03.729139 30913 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1164 milliseconds
14:06:03.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.729301 30905 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:06:03.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.730659 30905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:06:03.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.733513 30905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:06:03.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.734946 30905 hybrid_clock.cc:648] HybridClock initialized: now 1783519563734889 us; error 64 us; skew 500 ppm
14:06:03.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.735898 30905 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:06:03.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:03.738622 30537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111) [suppressed 9 similar messages]
14:06:03.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.744174 30905 webserver.cc:533] Webserver started at http://127.29.94.193:35541/ using document root <none> and password file <none>
14:06:03.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.745260 30905 fs_manager.cc:362] Metadata directory not provided
14:06:03.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.745520 30905 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:06:03.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.753973 30905 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
14:06:03.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.759188 30927 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:06:03.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.760480 30905 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
14:06:03.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.760916 30905 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e1649b01489046e3a63ca563aad7869c"
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:51 on dist-test-slave-59nb"
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.762873 30905 fs_report.cc:389] FS layout report
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/data
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:06:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:06:03.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:06:03.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:06:03.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:06:03.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.814446 30905 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:06:03.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.816104 30905 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:06:03.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.816586 30905 kserver.cc:163] Server-wide thread pool size limit: 3276
14:06:03.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.819362 30905 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:06:03.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.825557 30934 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:06:03.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.835129 30905 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:06:03.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.835399 30905 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.002s	sys 0.000s
14:06:03.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.835739 30905 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:06:03.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.840723 30905 ts_tablet_manager.cc:616] Registered 1 tablets
14:06:03.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.840921 30905 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
14:06:03.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.841526 30934 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap starting.
14:06:03.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.911005 30934 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Log is configured to *not* fsync() on all Append() calls
14:06:03.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.939774 30934 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:06:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.940804 30934 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap complete.
14:06:03.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.942205 30934 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Time spent bootstrapping tablet: real 0.101s	user 0.074s	sys 0.020s
14:06:03.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.962069 30934 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:03.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.964814 30934 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1649b01489046e3a63ca563aad7869c, State: Initialized, Role: FOLLOWER
14:06:03.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.966342 30934 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:03.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:03.970845 30934 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Time spent starting tablet: real 0.028s	user 0.029s	sys 0.000s
14:06:04.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.025552 30905 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.193:46741
14:06:04.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.026208 31042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.193:46741 every 8 connection(s)
14:06:04.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.028355 30905 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:04.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.036679 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 30905
14:06:04.037 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:04.063 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:06:04.064 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.94.194:43301
14:06:04.065 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:06:04.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.066591 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 30516
14:06:04.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.086695 31044 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:06:04.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.087296 31044 heartbeater.cc:461] Registering TS with master...
14:06:04.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.088447 31043 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:06:04.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.088666 31044 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:06:04.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.088799 31043 heartbeater.cc:461] Registering TS with master...
14:06:04.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.089721 31043 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:06:04.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.093276 30274 ts_manager.cc:194] Re-registered known tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:06:04.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.093497 30196 ts_manager.cc:194] Re-registered known tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:06:04.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.096292 30274 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.193:52679
14:06:04.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.097978 31045 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:06:04.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.098505 31045 heartbeater.cc:461] Registering TS with master...
14:06:04.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.099231 31045 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:06:04.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:04.102747 30121 ts_manager.cc:194] Re-registered known tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:06:04.109 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] handling channelInactive
14:06:04.110 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state READY due to: connection closed
14:06:04.110 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:04.111 [DEBUG - Time-limited test] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:04.113 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.113 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.113 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.121 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:04.121 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:04.121 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [3ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [3ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1085645738(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@1a5d8426 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a18f724 -> wakeup thread Time-limited test)) after lookup
14:06:04.122 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:04.124 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.124 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.124 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.141 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:04.141 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:04.142 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [3ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [3ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [12ms] sending RPC to server 957569291a7c402594d391a83985d059, [14ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [14ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1085645738(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@1a5d8426 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@705106336 after retry RPC@2012963347), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a18f724 -> wakeup thread Time-limited test -> (continuation of Deferred@705106336 after retry RPC@2012963347))) after lookup
14:06:04.143 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:04.144 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.144 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.144 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.181 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:04.182 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:04.182 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=71), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [3ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [3ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [12ms] sending RPC to server 957569291a7c402594d391a83985d059, [14ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [14ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [32ms] sending RPC to server 957569291a7c402594d391a83985d059, [34ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [34ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [71ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1085645738(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@1a5d8426 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@705106336 after retry RPC@2012963347) -> (continuation of Deferred@571114037 after retry RPC@239966109), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a18f724 -> wakeup thread Time-limited test -> (continuation of Deferred@705106336 after retry RPC@2012963347) -> (continuation of Deferred@571114037 after retry RPC@239966109))) after lookup
14:06:04.184 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:04.186 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.186 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.186 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.301 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:04.301 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:04.302 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=191), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [3ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [3ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [12ms] sending RPC to server 957569291a7c402594d391a83985d059, [14ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [14ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [32ms] sending RPC to server 957569291a7c402594d391a83985d059, [34ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [34ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [71ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [73ms] sending RPC to server 957569291a7c402594d391a83985d059, [76ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [76ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [190ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1085645738(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@1a5d8426 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@705106336 after retry RPC@2012963347) -> (continuation of Deferred@571114037 after retry RPC@239966109) -> (continuation of Deferred@1995436365 after retry RPC@674457353), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a18f724 -> wakeup thread Time-limited test -> (continuation of Deferred@705106336 after retry RPC@2012963347) -> (continuation of Deferred@571114037 after retry RPC@239966109) -> (continuation of Deferred@1995436365 after retry RPC@674457353))) after lookup
14:06:04.302 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:04.304 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.304 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.304 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.701 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:04.701 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:04.702 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=591), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [3ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [3ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [12ms] sending RPC to server 957569291a7c402594d391a83985d059, [14ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [14ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [32ms] sending RPC to server 957569291a7c402594d391a83985d059, [34ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [34ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [71ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [73ms] sending RPC to server 957569291a7c402594d391a83985d059, [76ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [76ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [190ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [192ms] sending RPC to server 957569291a7c402594d391a83985d059, [194ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [194ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [586ms] refreshing cache from master, [586ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [590ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1085645738(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@1a5d8426 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@705106336 after retry RPC@2012963347) -> (continuation of Deferred@571114037 after retry RPC@239966109) -> (continuation of Deferred@1995436365 after retry RPC@674457353) -> (continuation of Deferred@2080807085 after retry RPC@31135143), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a18f724 -> wakeup thread Time-limited test -> (continuation of Deferred@705106336 after retry RPC@2012963347) -> (continuation of Deferred@571114037 after retry RPC@239966109) -> (continuation of Deferred@1995436365 after retry RPC@674457353) -> (continuation of Deferred@2080807085 after retry RPC@31135143))) after lookup
14:06:04.703 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:04.704 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.704 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:04.704 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:05.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.099951 31043 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:06:05.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.274890 31054 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:06:05.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.275411 31054 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:05.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.277882 31054 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:05.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:05.283011 30931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:05.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:05.287737 30931 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:05.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.291061 30741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" is_pre_election: true
14:06:05.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.291862 31059 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 957569291a7c402594d391a83985d059)
14:06:05.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.292332 31059 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:05.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.293282 30741 raft_consensus.cc:2368] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e1649b01489046e3a63ca563aad7869c for earlier term 2. Current term is 3.
14:06:05.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.294462 31059 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:05.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.299573 30931 leader_election.cc:400] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 2 pre-election: Vote denied by peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127) with higher term. Message: Invalid argument: T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e1649b01489046e3a63ca563aad7869c for earlier term 2. Current term is 3.
14:06:05.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.299861 30931 leader_election.cc:403] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
14:06:05.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:05.300397 30677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111) [suppressed 1 similar messages]
14:06:05.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.300549 31054 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 1 FOLLOWER]: Advancing to term 3
14:06:05.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:05.305557 30677 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:05.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.308650 30997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "e1649b01489046e3a63ca563aad7869c" is_pre_election: true
14:06:05.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.309890 31054 raft_consensus.cc:2749] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127) with higher term. Message: Invalid argument: T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e1649b01489046e3a63ca563aad7869c for earlier term 2. Current term is 3.
14:06:05.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.310596 30997 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c7b9c0f64b41c6b3f9a0e3084cf3e8 in term 3.
14:06:05.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.315282 30677 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e1649b01489046e3a63ca563aad7869c, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: 957569291a7c402594d391a83985d059
14:06:05.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.316155 31059 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 3 FOLLOWER]: Leader pre-election won for term 4
14:06:05.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.316536 31059 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 957569291a7c402594d391a83985d059)
14:06:05.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.316847 31059 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 3 FOLLOWER]: Advancing to term 4
14:06:05.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.322384 31059 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:05.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.323905 31059 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 4 election: Requested vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:05.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.324759 30997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "e1649b01489046e3a63ca563aad7869c"
14:06:05.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.325228 30997 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 3 FOLLOWER]: Advancing to term 4
14:06:05.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:05.327468 30677 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:05.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.330443 30997 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c7b9c0f64b41c6b3f9a0e3084cf3e8 in term 4.
14:06:05.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.331387 30677 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e1649b01489046e3a63ca563aad7869c, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: 957569291a7c402594d391a83985d059
14:06:05.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.332031 31059 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 4 FOLLOWER]: Leader election won for term 4
14:06:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.333806 31059 raft_consensus.cc:697] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 4 LEADER]: Becoming Leader. State: Replica: e2c7b9c0f64b41c6b3f9a0e3084cf3e8, State: Running, Role: LEADER
14:06:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.334971 31059 consensus_queue.cc:237] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:05.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.345125 30274 catalog_manager.cc:5697] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 reported cstate change: term changed from 3 to 4, leader changed from 957569291a7c402594d391a83985d059 (127.29.94.194) to e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195). New cstate: current_term: 4 leader_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } health_report { overall_health: UNKNOWN } } }
14:06:05.562 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:05.563 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:05.563 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1453), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [3ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [3ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [12ms] sending RPC to server 957569291a7c402594d391a83985d059, [14ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [14ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [32ms] sending RPC to server 957569291a7c402594d391a83985d059, [34ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [34ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [71ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [73ms] sending RPC to server 957569291a7c402594d391a83985d059, [76ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [76ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [190ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [192ms] sending RPC to server 957569291a7c402594d391a83985d059, [194ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [194ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [586ms] refreshing cache from master, [586ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [590ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [592ms] sending RPC to server 957569291a7c402594d391a83985d059, [594ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [594ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [1446ms] refreshing cache from master, [1447ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [1452ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1085645738(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@1a5d8426 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@705106336 after retry RPC@2012963347) -> (continuation of Deferred@571114037 after retry RPC@239966109) -> (continuation of Deferred@1995436365 after retry RPC@674457353) -> (continuation of Deferred@2080807085 after retry RPC@31135143) -> (continuation of Deferred@1004809056 after retry RPC@1649108886), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1a18f724 -> wakeup thread Time-limited test -> (continuation of Deferred@705106336 after retry RPC@2012963347) -> (continuation of Deferred@571114037 after retry RPC@239966109) -> (continuation of Deferred@1995436365 after retry RPC@674457353) -> (continuation of Deferred@2080807085 after retry RPC@31135143) -> (continuation of Deferred@1004809056 after retry RPC@1649108886))) after lookup
14:06:05.565 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:05.566 [DEBUG - kudu-nio-2] (Connection.java:757) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] Successfully connected to peer
14:06:05.584 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0x94e7eea5, L:/127.0.0.1:33658 - R:/127.29.94.195:42127] using TOKEN/null
14:06:05.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.594728 30997 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 4 FOLLOWER]: Refusing update from remote peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
14:06:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:05.596923 30677 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 -> Peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Couldn't send request to peer 957569291a7c402594d391a83985d059. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:06:05.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.597951 31064 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
14:06:05.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.638720 31079 mvcc.cc:204] Tried to move back new op lower bound from 7305296140659339264 to 7305296139619979264. Current Snapshot: MvccSnapshot[applied={T|T < 7305296140659339264 or (T in {7305296140659339264})}]
14:06:05.665 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.94.194:43301
14:06:05.666 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:05.668010 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/logs
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:05.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.194:43301
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.194
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33225
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.194
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:05.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:06:05.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:05.995255 31086 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:06:05.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:05.995806 31086 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:06:05.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:05.996377 31086 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:06:06.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:06.040822 31086 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:06:06.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:06.042141 31086 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.194
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:06.092672 31086 tablet_server_runner.cc:78] Tablet server non-default flags:
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.194:43301
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.194
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33225
14:06:06.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.194
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/logs
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:06:06.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:06.094446 31086 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:06:06.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:06.096491 31086 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:06:06.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:06.112248 31093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:07.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:07.516112 31092 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 31086
14:06:07.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:07.621775 31092 kernel_stack_watchdog.cc:198] Thread 31086 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
14:06:07.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:06:07.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:06:07.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:06:07.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:06:07.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:07.623193 31086 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.510s	user 0.581s	sys 0.872s
14:06:07.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:07.623584 31086 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.510s	user 0.581s	sys 0.872s
14:06:07.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:07.623838 31095 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1509 milliseconds
14:06:07.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:06.114441 31094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:07.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:07.625356 31096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:07.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.625411 31086 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:06:07.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.629402 31086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:06:07.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.631645 31086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:06:07.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.633021 31086 hybrid_clock.cc:648] HybridClock initialized: now 1783519567632969 us; error 72 us; skew 500 ppm
14:06:07.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.633824 31086 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:06:07.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.640475 31086 webserver.cc:533] Webserver started at http://127.29.94.194:33225/ using document root <none> and password file <none>
14:06:07.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.641345 31086 fs_manager.cc:362] Metadata directory not provided
14:06:07.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.641563 31086 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:06:07.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.649518 31086 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
14:06:07.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.655324 31107 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:06:07.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.657756 31086 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.000s	sys 0.004s
14:06:07.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.658196 31086 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:07.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "957569291a7c402594d391a83985d059"
14:06:07.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:53 on dist-test-slave-59nb"
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.660956 31086 fs_report.cc:389] FS layout report
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/data
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:06:07.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:06:07.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:06:07.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.716259 31086 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:06:07.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.717978 31086 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:06:07.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.718484 31086 kserver.cc:163] Server-wide thread pool size limit: 3276
14:06:07.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.721417 31086 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:06:07.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.727523 31114 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:06:07.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.736521 31086 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:06:07.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.736776 31086 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.000s
14:06:07.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.737018 31086 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:06:07.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.741639 31086 ts_tablet_manager.cc:616] Registered 1 tablets
14:06:07.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.741832 31086 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.002s	sys 0.000s
14:06:07.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.742375 31114 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Bootstrap starting.
14:06:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.801652 31114 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Log is configured to *not* fsync() on all Append() calls
14:06:07.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.841300 31114 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:06:07.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.842235 31114 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Bootstrap complete.
14:06:07.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.843922 31114 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Time spent bootstrapping tablet: real 0.102s	user 0.054s	sys 0.046s
14:06:07.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.863147 31114 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:07.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.866035 31114 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957569291a7c402594d391a83985d059, State: Initialized, Role: FOLLOWER
14:06:07.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.867437 31114 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:07.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.872742 31114 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Time spent starting tablet: real 0.029s	user 0.022s	sys 0.006s
14:06:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.915951 31086 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.194:43301
14:06:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.916222 31222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.194:43301 every 8 connection(s)
14:06:07.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.918639 31086 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:06:07.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.927137 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 31086
14:06:07.928 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:07.958 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:06:07.959 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.94.195:42127
14:06:07.960 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:06:07.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.960357 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 30657
14:06:07.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.967871 31223 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:06:07.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.968393 31223 heartbeater.cc:461] Registering TS with master...
14:06:07.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.969198 31225 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:06:07.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.969547 31225 heartbeater.cc:461] Registering TS with master...
14:06:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.969890 31223 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:06:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.970654 31225 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:06:07.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.972457 31224 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:06:07.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.972832 31224 heartbeater.cc:461] Registering TS with master...
14:06:07.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.973703 31224 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:06:07.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.974093 30274 ts_manager.cc:194] Re-registered known tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:07.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.975337 30120 ts_manager.cc:194] Re-registered known tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:07.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.977147 30274 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.194:41711
14:06:07.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:07.977362 30196 ts_manager.cc:194] Re-registered known tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:07.997 [DEBUG - kudu-nio-2] (Connection.java:239) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] handling channelInactive
14:06:07.997 [DEBUG - kudu-nio-2] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state READY due to: connection closed
14:06:07.998 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:07.999 [DEBUG - Time-limited test] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:08.000 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.000 [DEBUG - kudu-nio-3] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.000 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.022 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:08.023 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:08.023 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1146372983(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@7d1aebbc -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@103c3212 -> wakeup thread Time-limited test)) after lookup
14:06:08.024 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:08.027 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.027 [DEBUG - kudu-nio-4] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.027 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.040 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:08.040 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:08.040 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [26ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [29ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [30ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [42ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1146372983(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@7d1aebbc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1451584034 after retry RPC@783391377), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@103c3212 -> wakeup thread Time-limited test -> (continuation of Deferred@1451584034 after retry RPC@783391377))) after lookup
14:06:08.041 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:08.042 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.042 [DEBUG - kudu-nio-5] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.042 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.079 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:08.080 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:08.080 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=82), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [26ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [29ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [30ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [42ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [43ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [44ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [44ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1146372983(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@7d1aebbc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1451584034 after retry RPC@783391377) -> (continuation of Deferred@549593162 after retry RPC@42288081), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@103c3212 -> wakeup thread Time-limited test -> (continuation of Deferred@1451584034 after retry RPC@783391377) -> (continuation of Deferred@549593162 after retry RPC@42288081))) after lookup
14:06:08.080 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:08.081 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.081 [DEBUG - kudu-nio-6] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.082 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.139 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:08.140 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:08.140 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=141), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [26ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [29ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [30ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [42ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [43ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [44ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [44ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [82ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [84ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [84ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [137ms] refreshing cache from master, [138ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [141ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1146372983(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@7d1aebbc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1451584034 after retry RPC@783391377) -> (continuation of Deferred@549593162 after retry RPC@42288081) -> (continuation of Deferred@985010979 after retry RPC@310723695), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@103c3212 -> wakeup thread Time-limited test -> (continuation of Deferred@1451584034 after retry RPC@783391377) -> (continuation of Deferred@549593162 after retry RPC@42288081) -> (continuation of Deferred@985010979 after retry RPC@310723695))) after lookup
14:06:08.140 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:08.141 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.141 [DEBUG - kudu-nio-7] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.142 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.239 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:08.240 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:08.240 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=242), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [26ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [29ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [30ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [42ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [43ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [44ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [44ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [82ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [84ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [84ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [137ms] refreshing cache from master, [138ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [141ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [142ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [143ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [144ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [238ms] refreshing cache from master, [238ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [241ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1146372983(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@7d1aebbc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1451584034 after retry RPC@783391377) -> (continuation of Deferred@549593162 after retry RPC@42288081) -> (continuation of Deferred@985010979 after retry RPC@310723695) -> (continuation of Deferred@613404147 after retry RPC@479281969), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@103c3212 -> wakeup thread Time-limited test -> (continuation of Deferred@1451584034 after retry RPC@783391377) -> (continuation of Deferred@549593162 after retry RPC@42288081) -> (continuation of Deferred@985010979 after retry RPC@310723695) -> (continuation of Deferred@613404147 after retry RPC@479281969))) after lookup
14:06:08.241 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:08.242 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.242 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.242 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:08.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:08.980854 31223 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:06:09.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.083528 31234 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e2c7b9c0f64b41c6b3f9a0e3084cf3e8)
14:06:09.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.083927 31234 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:09.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.085373 31234 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:09.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:09.091934 30931 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127): Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111)
14:06:09.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.098088 31177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "957569291a7c402594d391a83985d059" is_pre_election: true
14:06:09.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.098851 31177 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1649b01489046e3a63ca563aad7869c in term 3.
14:06:09.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.100063 30931 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e1649b01489046e3a63ca563aad7869c; no voters: e2c7b9c0f64b41c6b3f9a0e3084cf3e8
14:06:09.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.100744 31234 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 4 FOLLOWER]: Leader pre-election won for term 5
14:06:09.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.101060 31234 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 4 FOLLOWER]: Starting leader election (detected failure of leader e2c7b9c0f64b41c6b3f9a0e3084cf3e8)
14:06:09.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.101399 31234 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 4 FOLLOWER]: Advancing to term 5
14:06:09.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.106144 31234 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:09.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.107744 31234 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 5 election: Requested vote from peers e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:09.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.109241 31177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "957569291a7c402594d391a83985d059"
14:06:09.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.109673 31177 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 3 FOLLOWER]: Advancing to term 5
14:06:09.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:09.111217 30931 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127): Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111)
14:06:09.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.116892 31177 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1649b01489046e3a63ca563aad7869c in term 5.
14:06:09.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.117808 30931 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e1649b01489046e3a63ca563aad7869c; no voters: e2c7b9c0f64b41c6b3f9a0e3084cf3e8
14:06:09.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.118410 31234 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 5 FOLLOWER]: Leader election won for term 5
14:06:09.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.119539 31234 raft_consensus.cc:697] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 5 LEADER]: Becoming Leader. State: Replica: e1649b01489046e3a63ca563aad7869c, State: Running, Role: LEADER
14:06:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.120397 31234 consensus_queue.cc:237] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:09.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.129294 30274 catalog_manager.cc:5697] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c reported cstate change: term changed from 4 to 5, leader changed from e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195) to e1649b01489046e3a63ca563aad7869c (127.29.94.193). New cstate: current_term: 5 leader_uuid: "e1649b01489046e3a63ca563aad7869c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } health_report { overall_health: UNKNOWN } } }
14:06:09.180 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:09.180 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:09.181 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1182), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [26ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [29ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [30ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [42ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [43ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [44ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [44ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [82ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [84ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [84ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [137ms] refreshing cache from master, [138ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [141ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [142ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [143ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [144ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [238ms] refreshing cache from master, [238ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [241ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [243ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [244ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [244ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [1178ms] refreshing cache from master, [1178ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [1182ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1146372983(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@7d1aebbc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1451584034 after retry RPC@783391377) -> (continuation of Deferred@549593162 after retry RPC@42288081) -> (continuation of Deferred@985010979 after retry RPC@310723695) -> (continuation of Deferred@613404147 after retry RPC@479281969) -> (continuation of Deferred@1242594876 after retry RPC@314331810), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@103c3212 -> wakeup thread Time-limited test -> (continuation of Deferred@1451584034 after retry RPC@783391377) -> (continuation of Deferred@549593162 after retry RPC@42288081) -> (continuation of Deferred@985010979 after retry RPC@310723695) -> (continuation of Deferred@613404147 after retry RPC@479281969) -> (continuation of Deferred@1242594876 after retry RPC@314331810))) after lookup
14:06:09.182 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:09.183 [DEBUG - kudu-nio-1] (Connection.java:757) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] Successfully connected to peer
14:06:09.207 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x0f280121, L:/127.0.0.1:60168 - R:/127.29.94.193:46741] using TOKEN/null
14:06:09.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.219271 31177 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 5 FOLLOWER]: Refusing update from remote peer e1649b01489046e3a63ca563aad7869c: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 14. (index mismatch)
14:06:09.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.221082 31239 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [LEADER]: Connected to new peer: Peer: permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
14:06:09.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:09.221691 30931 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c -> Peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127): Couldn't send request to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:06:09.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.237144 31256 mvcc.cc:204] Tried to move back new op lower bound from 7305296140659339264 to 7305296139619979264. Current Snapshot: MvccSnapshot[applied={T|T < 7305296140659339264}]
14:06:09.287 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.94.195:42127
14:06:09.287 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.288695 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/logs
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.195:42127
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.195
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35565
14:06:09.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.195
14:06:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:06:09.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:09.578871 31263 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:06:09.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:09.579391 31263 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:06:09.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:09.580119 31263 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:06:09.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:09.612282 31263 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:06:09.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:09.613288 31263 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.195
14:06:09.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.649178 31263 tablet_server_runner.cc:78] Tablet server non-default flags:
14:06:09.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:09.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:09.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.195:42127
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.195
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35565
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.195
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/logs
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:06:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:06:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:06:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:06:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:06:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:06:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.650905 31263 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:06:09.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:09.652629 31263 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:06:09.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:09.664773 31269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:10.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:10.326606 30931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111) [suppressed 4 similar messages]
14:06:11.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:11.067754 31268 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 31263
14:06:11.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:11.136344 31268 kernel_stack_watchdog.cc:198] Thread 31263 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
14:06:11.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:06:11.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:06:11.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:06:11.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:06:11.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:09.665324 31270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:11.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:11.139026 31263 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.474s	user 0.492s	sys 0.930s
14:06:11.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:11.139456 31263 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.475s	user 0.492s	sys 0.930s
14:06:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:11.142127 31272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:11.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:11.145337 31271 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1474 milliseconds
14:06:11.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.145367 31263 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:06:11.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.146610 31263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:06:11.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.148717 31263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:06:11.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.150100 31263 hybrid_clock.cc:648] HybridClock initialized: now 1783519571150045 us; error 67 us; skew 500 ppm
14:06:11.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.151130 31263 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:06:11.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.157426 31263 webserver.cc:533] Webserver started at http://127.29.94.195:35565/ using document root <none> and password file <none>
14:06:11.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.158344 31263 fs_manager.cc:362] Metadata directory not provided
14:06:11.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.158607 31263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:06:11.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.167438 31263 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.004s	sys 0.003s
14:06:11.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.172130 31282 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:06:11.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.173485 31263 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
14:06:11.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.173867 31263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:54 on dist-test-slave-59nb"
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.175945 31263 fs_report.cc:389] FS layout report
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/data
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:06:11.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:06:11.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:06:11.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.230098 31263 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:06:11.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.231801 31263 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:06:11.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.232214 31263 kserver.cc:163] Server-wide thread pool size limit: 3276
14:06:11.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.235020 31263 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:06:11.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.242064 31289 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:06:11.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.250324 31263 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:06:11.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.250643 31263 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
14:06:11.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.251015 31263 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:06:11.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.256557 31263 ts_tablet_manager.cc:616] Registered 1 tablets
14:06:11.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.256819 31263 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.002s	sys 0.002s
14:06:11.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.257308 31289 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Bootstrap starting.
14:06:11.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.318693 31289 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Log is configured to *not* fsync() on all Append() calls
14:06:11.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.360628 31289 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:06:11.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.361624 31289 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Bootstrap complete.
14:06:11.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.363073 31289 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Time spent bootstrapping tablet: real 0.106s	user 0.065s	sys 0.038s
14:06:11.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.381654 31289 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:11.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.384578 31289 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c7b9c0f64b41c6b3f9a0e3084cf3e8, State: Initialized, Role: FOLLOWER
14:06:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.385782 31289 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:11.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.389524 31289 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Time spent starting tablet: real 0.026s	user 0.017s	sys 0.008s
14:06:11.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.449031 31263 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.195:42127
14:06:11.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.449258 31397 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.195:42127 every 8 connection(s)
14:06:11.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.451697 31263 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:06:11.464 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:11.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.461908 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 31263
14:06:11.515 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:06:11.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.520542 31399 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:06:11.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.521132 31399 heartbeater.cc:461] Registering TS with master...
14:06:11.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.522547 31399 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:06:11.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.527289 31398 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:06:11.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.527696 31398 heartbeater.cc:461] Registering TS with master...
14:06:11.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.527637 30195 ts_manager.cc:194] Re-registered known tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:11.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.528653 31398 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:06:11.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.532152 30276 ts_manager.cc:194] Re-registered known tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:11.533 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.94.193:46741
14:06:11.533 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:06:11.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.534070 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 30905
14:06:11.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.536808 30276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.195:58311
14:06:11.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.550101 31400 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:06:11.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.550465 31400 heartbeater.cc:461] Registering TS with master...
14:06:11.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.551115 31400 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:06:11.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:11.555703 30121 ts_manager.cc:194] Re-registered known tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:11.568 [DEBUG - kudu-nio-1] (Connection.java:239) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] handling channelInactive
14:06:11.568 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:11.568 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state READY due to: connection closed
14:06:11.569 [DEBUG - Time-limited test] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:11.570 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.570 [DEBUG - kudu-nio-2] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.570 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.580 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:11.581 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:11.581 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1066848419(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@4f4facb1 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@253cd93f -> wakeup thread Time-limited test)) after lookup
14:06:11.582 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:11.583 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.583 [DEBUG - kudu-nio-3] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.583 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.600 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:11.600 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:11.601 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [15ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [15ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [32ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1066848419(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@4f4facb1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@487438466 after retry RPC@1589342013), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@253cd93f -> wakeup thread Time-limited test -> (continuation of Deferred@487438466 after retry RPC@1589342013))) after lookup
14:06:11.601 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:11.603 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.603 [DEBUG - kudu-nio-4] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.603 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.641 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:11.641 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:11.642 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [15ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [15ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [32ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [33ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [35ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [35ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [68ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [73ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1066848419(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@4f4facb1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@487438466 after retry RPC@1589342013) -> (continuation of Deferred@1899658660 after retry RPC@1954358841), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@253cd93f -> wakeup thread Time-limited test -> (continuation of Deferred@487438466 after retry RPC@1589342013) -> (continuation of Deferred@1899658660 after retry RPC@1954358841))) after lookup
14:06:11.643 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:11.645 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.645 [DEBUG - kudu-nio-5] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.645 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.681 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:11.682 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:11.682 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=114), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [15ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [15ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [32ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [33ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [35ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [35ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [68ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [73ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [75ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [77ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [77ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [108ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1066848419(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@4f4facb1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@487438466 after retry RPC@1589342013) -> (continuation of Deferred@1899658660 after retry RPC@1954358841) -> (continuation of Deferred@281697430 after retry RPC@219933784), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@253cd93f -> wakeup thread Time-limited test -> (continuation of Deferred@487438466 after retry RPC@1589342013) -> (continuation of Deferred@1899658660 after retry RPC@1954358841) -> (continuation of Deferred@281697430 after retry RPC@219933784))) after lookup
14:06:11.683 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:11.685 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.685 [DEBUG - kudu-nio-6] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.685 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.821 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:11.821 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:11.821 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=253), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [15ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [15ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [32ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [33ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [35ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [35ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [68ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [73ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [75ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [77ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [77ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [108ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [115ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [117ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [117ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [248ms] refreshing cache from master, [248ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [252ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1066848419(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@4f4facb1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@487438466 after retry RPC@1589342013) -> (continuation of Deferred@1899658660 after retry RPC@1954358841) -> (continuation of Deferred@281697430 after retry RPC@219933784) -> (continuation of Deferred@772861802 after retry RPC@2092675036), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@253cd93f -> wakeup thread Time-limited test -> (continuation of Deferred@487438466 after retry RPC@1589342013) -> (continuation of Deferred@1899658660 after retry RPC@1954358841) -> (continuation of Deferred@281697430 after retry RPC@219933784) -> (continuation of Deferred@772861802 after retry RPC@2092675036))) after lookup
14:06:11.822 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:11.824 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.824 [DEBUG - kudu-nio-7] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:11.824 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:12.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.543095 31398 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:06:12.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.709420 31410 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1649b01489046e3a63ca563aad7869c)
14:06:12.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.709954 31410 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:12.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.714237 31410 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:12.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:12.717550 31111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:06:12.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:12.724730 31111 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:06:12.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.761732 31352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" is_pre_election: true
14:06:12.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.762526 31352 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957569291a7c402594d391a83985d059 in term 4.
14:06:12.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.763883 31111 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: e1649b01489046e3a63ca563aad7869c
14:06:12.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.764604 31410 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 5 FOLLOWER]: Leader pre-election won for term 6
14:06:12.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.764912 31410 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e1649b01489046e3a63ca563aad7869c)
14:06:12.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.765141 31410 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 5 FOLLOWER]: Advancing to term 6
14:06:12.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.769476 31410 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:12.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.770991 31410 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 6 election: Requested vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:12.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.772647 31352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:06:12.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.773190 31352 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 4 FOLLOWER]: Advancing to term 6
14:06:12.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:12.774426 31111 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:06:12.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.780090 31352 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957569291a7c402594d391a83985d059 in term 6.
14:06:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.781095 31111 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: e1649b01489046e3a63ca563aad7869c
14:06:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.781714 31410 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 6 FOLLOWER]: Leader election won for term 6
14:06:12.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.783437 31410 raft_consensus.cc:697] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 6 LEADER]: Becoming Leader. State: Replica: 957569291a7c402594d391a83985d059, State: Running, Role: LEADER
14:06:12.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.784387 31410 consensus_queue.cc:237] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:12.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:12.793711 30276 catalog_manager.cc:5697] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 reported cstate change: term changed from 5 to 6, leader changed from e1649b01489046e3a63ca563aad7869c (127.29.94.193) to 957569291a7c402594d391a83985d059 (127.29.94.194). New cstate: current_term: 6 leader_uuid: "957569291a7c402594d391a83985d059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } health_report { overall_health: HEALTHY } } }
14:06:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:13.184860 31352 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 6 FOLLOWER]: Refusing update from remote peer 957569291a7c402594d391a83985d059: Log matching property violated. Preceding OpId in replica: term: 4 index: 12. Preceding OpId from leader: term: 6 index: 17. (index mismatch)
14:06:13.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:13.186486 31410 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
14:06:13.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:13.225857 31111 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 -> Peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Couldn't send request to peer e1649b01489046e3a63ca563aad7869c. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:06:13.501 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:13.501 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:13.502 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1933), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [15ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [15ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [32ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [33ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [35ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [35ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [68ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [73ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [75ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [77ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [77ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [108ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [113ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [115ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [117ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [117ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [248ms] refreshing cache from master, [248ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [252ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [254ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [256ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [256ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [1928ms] refreshing cache from master, [1928ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [1932ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1066848419(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@4f4facb1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@487438466 after retry RPC@1589342013) -> (continuation of Deferred@1899658660 after retry RPC@1954358841) -> (continuation of Deferred@281697430 after retry RPC@219933784) -> (continuation of Deferred@772861802 after retry RPC@2092675036) -> (continuation of Deferred@210219259 after retry RPC@769862648), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@253cd93f -> wakeup thread Time-limited test -> (continuation of Deferred@487438466 after retry RPC@1589342013) -> (continuation of Deferred@1899658660 after retry RPC@1954358841) -> (continuation of Deferred@281697430 after retry RPC@219933784) -> (continuation of Deferred@772861802 after retry RPC@2092675036) -> (continuation of Deferred@210219259 after retry RPC@769862648))) after lookup
14:06:13.503 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:13.504 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] Successfully connected to peer
14:06:13.524 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x05b4ac99, L:/127.0.0.1:54414 - R:/127.29.94.194:43301] using TOKEN/null
14:06:13.567 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.94.193:46741
14:06:13.568 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:06:13.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:13.569448 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:13.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:13.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/logs
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.193:46741
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.193
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35541
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.193
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:13.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:13.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:13.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:06:13.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:13.746944 31394 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfb00 after lost signal to thread 31264
14:06:13.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:13.748260 31394 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfca0 after lost signal to thread 31397
14:06:13.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:13.868826 31438 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:06:13.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:13.869462 31438 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:06:13.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:13.870191 31438 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:06:13.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:13.921108 31438 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:06:13.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:13.922077 31438 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.193
14:06:13.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:13.960163 31438 tablet_server_runner.cc:78] Tablet server non-default flags:
14:06:13.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:13.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:13.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:13.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:06:13.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.193:46741
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.193
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35541
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.193
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/logs
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:06:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:06:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:06:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:13.961865 31438 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:06:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:13.963655 31438 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:06:13.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:13.978953 31447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:15.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:13.987411 31446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:15.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:15.369544 31449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:15.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:15.371791 31438 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.391s	user 0.535s	sys 0.838s
14:06:15.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:15.372386 31438 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.392s	user 0.536s	sys 0.839s
14:06:15.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.372860 31438 server_base.cc:1061] running on GCE node
14:06:15.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.374958 31438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:06:15.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.377913 31438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:06:15.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.379612 31438 hybrid_clock.cc:648] HybridClock initialized: now 1783519575379582 us; error 71 us; skew 500 ppm
14:06:15.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.380715 31438 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:06:15.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.389849 31438 webserver.cc:533] Webserver started at http://127.29.94.193:35541/ using document root <none> and password file <none>
14:06:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.391744 31438 fs_manager.cc:362] Metadata directory not provided
14:06:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.392232 31438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:06:15.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.405670 31438 fs_manager.cc:714] Time spent opening directory manager: real 0.009s	user 0.003s	sys 0.006s
14:06:15.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.414446 31459 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:06:15.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.415970 31438 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.002s	sys 0.003s
14:06:15.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.416380 31438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:15.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e1649b01489046e3a63ca563aad7869c"
14:06:15.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:51 on dist-test-slave-59nb"
14:06:15.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.419113 31438 fs_report.cc:389] FS layout report
14:06:15.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:06:15.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:15.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:15.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/data
14:06:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:06:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:06:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:06:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:06:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:06:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:06:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:06:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:06:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:06:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:06:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:06:15.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.519646 31438 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:06:15.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.521072 31438 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:06:15.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.521466 31438 kserver.cc:163] Server-wide thread pool size limit: 3276
14:06:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.524359 31438 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:06:15.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.530903 31466 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:06:15.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.538872 31438 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:06:15.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.539093 31438 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
14:06:15.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.539309 31438 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:06:15.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.544111 31438 ts_tablet_manager.cc:616] Registered 1 tablets
14:06:15.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.544368 31438 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
14:06:15.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.544723 31466 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap starting.
14:06:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.622875 31466 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Log is configured to *not* fsync() on all Append() calls
14:06:15.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.700135 31466 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:06:15.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.701289 31466 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap complete.
14:06:15.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.703703 31466 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Time spent bootstrapping tablet: real 0.159s	user 0.107s	sys 0.043s
14:06:15.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.732038 31466 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:15.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.735603 31466 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1649b01489046e3a63ca563aad7869c, State: Initialized, Role: FOLLOWER
14:06:15.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.737231 31466 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:15.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.743722 31466 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Time spent starting tablet: real 0.040s	user 0.026s	sys 0.012s
14:06:15.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.744807 31438 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.193:46741
14:06:15.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.745272 31574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.193:46741 every 8 connection(s)
14:06:15.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.747964 31438 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:15.750 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:15.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.749418 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 31438
14:06:15.775 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:06:15.777 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.94.194:43301
14:06:15.778 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:06:15.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.778504 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 31086
14:06:15.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.827931 31575 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:06:15.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.828533 31575 heartbeater.cc:461] Registering TS with master...
14:06:15.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.830016 31575 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:06:15.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.833155 31576 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:06:15.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.833523 31576 heartbeater.cc:461] Registering TS with master...
14:06:15.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.834474 31576 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:06:15.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.839247 30195 ts_manager.cc:194] Re-registered known tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:06:15.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.839900 30276 ts_manager.cc:194] Re-registered known tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:06:15.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.842690 30276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.193:50899
14:06:15.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.856145 31580 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:06:15.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.856528 31580 heartbeater.cc:461] Registering TS with master...
14:06:15.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.857183 31580 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:06:15.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:15.861214 30121 ts_manager.cc:194] Re-registered known tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:06:15.866 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] handling channelInactive
14:06:15.866 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state READY due to: connection closed
14:06:15.868 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:15.869 [DEBUG - Time-limited test] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:15.870 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:15.871 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:15.871 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:15.881 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:15.881 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:15.882 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [2ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [11ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2048250253(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@bc64d1c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@cefb3d1 -> wakeup thread Time-limited test)) after lookup
14:06:15.883 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:15.884 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:15.884 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:15.884 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:15.900 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:15.900 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:15.901 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [2ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [11ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server 957569291a7c402594d391a83985d059, [15ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [15ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2048250253(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@bc64d1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@cefb3d1 -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193))) after lookup
14:06:15.901 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:15.903 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:15.903 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:15.903 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:15.941 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:15.942 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:15.942 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [2ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [11ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server 957569291a7c402594d391a83985d059, [15ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [15ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [32ms] sending RPC to server 957569291a7c402594d391a83985d059, [34ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [34ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [72ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2048250253(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@bc64d1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193) -> (continuation of Deferred@842712831 after retry RPC@1236738334), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@cefb3d1 -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193) -> (continuation of Deferred@842712831 after retry RPC@1236738334))) after lookup
14:06:15.943 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:15.944 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:15.945 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:15.945 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:16.060 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:16.061 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:16.061 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=192), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [2ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [11ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server 957569291a7c402594d391a83985d059, [15ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [15ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [32ms] sending RPC to server 957569291a7c402594d391a83985d059, [34ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [34ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [72ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [74ms] sending RPC to server 957569291a7c402594d391a83985d059, [76ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [76ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [191ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2048250253(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@bc64d1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193) -> (continuation of Deferred@842712831 after retry RPC@1236738334) -> (continuation of Deferred@2017005241 after retry RPC@141979505), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@cefb3d1 -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193) -> (continuation of Deferred@842712831 after retry RPC@1236738334) -> (continuation of Deferred@2017005241 after retry RPC@141979505))) after lookup
14:06:16.062 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:16.063 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:16.064 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:16.064 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:16.400 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:16.401 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:16.401 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=532), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [2ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [11ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server 957569291a7c402594d391a83985d059, [15ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [15ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [32ms] sending RPC to server 957569291a7c402594d391a83985d059, [34ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [34ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [72ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [74ms] sending RPC to server 957569291a7c402594d391a83985d059, [76ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [76ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [191ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [193ms] sending RPC to server 957569291a7c402594d391a83985d059, [195ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [195ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [527ms] refreshing cache from master, [528ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [531ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2048250253(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@bc64d1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193) -> (continuation of Deferred@842712831 after retry RPC@1236738334) -> (continuation of Deferred@2017005241 after retry RPC@141979505) -> (continuation of Deferred@2060570225 after retry RPC@1105673149), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@cefb3d1 -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193) -> (continuation of Deferred@842712831 after retry RPC@1236738334) -> (continuation of Deferred@2017005241 after retry RPC@141979505) -> (continuation of Deferred@2060570225 after retry RPC@1105673149))) after lookup
14:06:16.402 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:16.404 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:16.404 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:16.404 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:16.741 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:16.741 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:16.741 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=872), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [2ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [11ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server 957569291a7c402594d391a83985d059, [15ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [15ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [32ms] sending RPC to server 957569291a7c402594d391a83985d059, [34ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [34ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [72ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [74ms] sending RPC to server 957569291a7c402594d391a83985d059, [76ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [76ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [191ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [193ms] sending RPC to server 957569291a7c402594d391a83985d059, [195ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [195ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [527ms] refreshing cache from master, [528ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [531ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [533ms] sending RPC to server 957569291a7c402594d391a83985d059, [535ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [535ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [867ms] refreshing cache from master, [868ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [871ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2048250253(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@bc64d1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193) -> (continuation of Deferred@842712831 after retry RPC@1236738334) -> (continuation of Deferred@2017005241 after retry RPC@141979505) -> (continuation of Deferred@2060570225 after retry RPC@1105673149) -> (continuation of Deferred@2027653095 after retry RPC@1667474), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@cefb3d1 -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193) -> (continuation of Deferred@842712831 after retry RPC@1236738334) -> (continuation of Deferred@2017005241 after retry RPC@141979505) -> (continuation of Deferred@2060570225 after retry RPC@1105673149) -> (continuation of Deferred@2027653095 after retry RPC@1667474))) after lookup
14:06:16.742 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:16.744 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:16.744 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:16.744 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:16.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.846550 31575 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:06:16.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.877650 31588 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:06:16.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.878144 31588 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:16.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.881119 31588 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:16.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:16.885840 31463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:16.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:16.890488 31463 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:16.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.894152 31352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" is_pre_election: true
14:06:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.895648 31463 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1649b01489046e3a63ca563aad7869c; no voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8
14:06:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.896479 31588 raft_consensus.cc:2749] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
14:06:16.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.965327 31593 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 957569291a7c402594d391a83985d059)
14:06:16.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.965726 31593 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:16.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.967947 31593 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:16.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:16.972098 31286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:16.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:16.976449 31286 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:16.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.981117 31529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "e1649b01489046e3a63ca563aad7869c" is_pre_election: true
14:06:16.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.981864 31529 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c7b9c0f64b41c6b3f9a0e3084cf3e8 in term 5.
14:06:16.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.983224 31286 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e1649b01489046e3a63ca563aad7869c, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: 957569291a7c402594d391a83985d059
14:06:16.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.983944 31593 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 6 FOLLOWER]: Leader pre-election won for term 7
14:06:16.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.984316 31593 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 957569291a7c402594d391a83985d059)
14:06:16.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.984647 31593 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 6 FOLLOWER]: Advancing to term 7
14:06:16.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.991112 31593 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:16.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.992920 31593 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 7 election: Requested vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:16.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.993713 31529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "e1649b01489046e3a63ca563aad7869c"
14:06:16.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:16.994144 31529 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 5 FOLLOWER]: Advancing to term 7
14:06:17.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:16.999680 31286 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:17.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:17.002085 31529 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c7b9c0f64b41c6b3f9a0e3084cf3e8 in term 7.
14:06:17.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:17.003171 31286 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e1649b01489046e3a63ca563aad7869c, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: 957569291a7c402594d391a83985d059
14:06:17.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:17.004040 31593 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 7 FOLLOWER]: Leader election won for term 7
14:06:17.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:17.005971 31593 raft_consensus.cc:697] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 7 LEADER]: Becoming Leader. State: Replica: e2c7b9c0f64b41c6b3f9a0e3084cf3e8, State: Running, Role: LEADER
14:06:17.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:17.007144 31593 consensus_queue.cc:237] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:17.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:17.016932 30276 catalog_manager.cc:5697] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 reported cstate change: term changed from 6 to 7, leader changed from 957569291a7c402594d391a83985d059 (127.29.94.194) to e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195). New cstate: current_term: 7 leader_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } health_report { overall_health: UNKNOWN } } }
14:06:17.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:17.468773 31529 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 7 FOLLOWER]: Refusing update from remote peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Log matching property violated. Preceding OpId in replica: term: 5 index: 16. Preceding OpId from leader: term: 7 index: 21. (index mismatch)
14:06:17.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:17.470186 31593 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
14:06:17.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:17.484715 31611 mvcc.cc:204] Tried to move back new op lower bound from 7305296173174435840 to 7305296170133241856. Current Snapshot: MvccSnapshot[applied={T|T < 7305296173174435840}]
14:06:17.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:17.508105 31286 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 -> Peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Couldn't send request to peer 957569291a7c402594d391a83985d059. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:06:20.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:20.101150 31286 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 -> Peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Couldn't send request to peer 957569291a7c402594d391a83985d059. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:06:20.600 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:20.601 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:20.601 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=14, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=4732), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [2ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [11ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server 957569291a7c402594d391a83985d059, [15ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [15ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [31ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [32ms] sending RPC to server 957569291a7c402594d391a83985d059, [34ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [34ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [72ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [74ms] sending RPC to server 957569291a7c402594d391a83985d059, [76ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [76ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [191ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [193ms] sending RPC to server 957569291a7c402594d391a83985d059, [195ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [195ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [527ms] refreshing cache from master, [528ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [531ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [533ms] sending RPC to server 957569291a7c402594d391a83985d059, [535ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [535ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [867ms] refreshing cache from master, [868ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [871ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [873ms] sending RPC to server 957569291a7c402594d391a83985d059, [875ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [875ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [4727ms] refreshing cache from master, [4727ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [4731ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2048250253(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@bc64d1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193) -> (continuation of Deferred@842712831 after retry RPC@1236738334) -> (continuation of Deferred@2017005241 after retry RPC@141979505) -> (continuation of Deferred@2060570225 after retry RPC@1105673149) -> (continuation of Deferred@2027653095 after retry RPC@1667474) -> (continuation of Deferred@1050325034 after retry RPC@1975829386), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@cefb3d1 -> wakeup thread Time-limited test -> (continuation of Deferred@1980750602 after retry RPC@118460193) -> (continuation of Deferred@842712831 after retry RPC@1236738334) -> (continuation of Deferred@2017005241 after retry RPC@141979505) -> (continuation of Deferred@2060570225 after retry RPC@1105673149) -> (continuation of Deferred@2027653095 after retry RPC@1667474) -> (continuation of Deferred@1050325034 after retry RPC@1975829386))) after lookup
14:06:20.602 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:20.603 [DEBUG - kudu-nio-0] (Connection.java:757) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] Successfully connected to peer
14:06:20.624 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x021b0cd7, L:/127.0.0.1:54266 - R:/127.29.94.195:42127] using TOKEN/null
14:06:20.686 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.94.194:43301
14:06:20.688 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:06:20.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:20.689565 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/logs
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.194:43301
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.194
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33225
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.194
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:06:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:21.033603 31636 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:06:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:21.034159 31636 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:06:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:21.034623 31636 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:06:21.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:21.071210 31636 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:06:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:21.072168 31636 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.194
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:21.109620 31636 tablet_server_runner.cc:78] Tablet server non-default flags:
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.194:43301
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:21.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.194
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33225
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.194
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/logs
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:06:21.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:06:21.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:06:21.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:06:21.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:21.111398 31636 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:06:21.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:21.113446 31636 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:06:21.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:21.126312 31643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:22.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:22.032310 31286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111) [suppressed 10 similar messages]
14:06:22.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:21.127908 31644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:22.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:22.268235 31646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:22.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:22.270485 31645 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1139 milliseconds
14:06:22.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.270635 31636 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:06:22.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.271966 31636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:06:22.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.274387 31636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:06:22.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.275732 31636 hybrid_clock.cc:648] HybridClock initialized: now 1783519582275676 us; error 66 us; skew 500 ppm
14:06:22.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.276557 31636 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:06:22.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.283283 31636 webserver.cc:533] Webserver started at http://127.29.94.194:33225/ using document root <none> and password file <none>
14:06:22.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.284343 31636 fs_manager.cc:362] Metadata directory not provided
14:06:22.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.284536 31636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:06:22.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.292593 31636 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.000s	sys 0.004s
14:06:22.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.298036 31658 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:06:22.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.299292 31636 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.004s
14:06:22.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.299615 31636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:22.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "957569291a7c402594d391a83985d059"
14:06:22.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:53 on dist-test-slave-59nb"
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.301822 31636 fs_report.cc:389] FS layout report
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/data
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:06:22.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:06:22.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.374532 31636 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:06:22.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.376197 31636 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:06:22.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.376638 31636 kserver.cc:163] Server-wide thread pool size limit: 3276
14:06:22.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.379545 31636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:06:22.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.385612 31665 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:06:22.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.394202 31636 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:06:22.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.394560 31636 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
14:06:22.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.394918 31636 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:06:22.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.400290 31636 ts_tablet_manager.cc:616] Registered 1 tablets
14:06:22.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.400490 31636 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.004s	sys 0.000s
14:06:22.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.401023 31665 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Bootstrap starting.
14:06:22.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.460669 31665 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Log is configured to *not* fsync() on all Append() calls
14:06:22.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:22.493057 31286 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 -> Peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Couldn't send request to peer 957569291a7c402594d391a83985d059. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
14:06:22.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.525677 31665 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:06:22.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.526913 31665 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Bootstrap complete.
14:06:22.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.528396 31665 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Time spent bootstrapping tablet: real 0.128s	user 0.086s	sys 0.038s
14:06:22.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.551014 31665 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:22.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.553996 31665 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957569291a7c402594d391a83985d059, State: Initialized, Role: FOLLOWER
14:06:22.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.555403 31665 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:22.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.560067 31665 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Time spent starting tablet: real 0.031s	user 0.025s	sys 0.006s
14:06:22.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.605338 31636 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.194:43301
14:06:22.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.605616 31773 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.194:43301 every 8 connection(s)
14:06:22.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.608347 31636 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:06:22.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.618206 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 31636
14:06:22.619 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:22.649 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:06:22.649 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.94.195:42127
14:06:22.650 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:06:22.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.647531 31775 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:06:22.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.648103 31775 heartbeater.cc:461] Registering TS with master...
14:06:22.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.650339 31775 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:06:22.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.650986 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 31263
14:06:22.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.658056 30195 ts_manager.cc:194] Re-registered known tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.658706 31774 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:06:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.659109 31774 heartbeater.cc:461] Registering TS with master...
14:06:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.660117 31774 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:06:22.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.663098 30276 ts_manager.cc:194] Re-registered known tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:22.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.664948 30276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.194:36127
14:06:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.669848 31776 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:06:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.670122 31776 heartbeater.cc:461] Registering TS with master...
14:06:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.670563 31776 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:06:22.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:22.673527 30121 ts_manager.cc:194] Re-registered known tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:22.687 [DEBUG - kudu-nio-0] (Connection.java:239) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] handling channelInactive
14:06:22.687 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:22.687 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state READY due to: connection closed
14:06:22.688 [DEBUG - Time-limited test] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:22.689 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.689 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.689 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.700 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:22.701 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:22.701 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [1ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [1ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2124463346(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@6d05b5cd -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3265d26d -> wakeup thread Time-limited test)) after lookup
14:06:22.702 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:22.703 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.703 [DEBUG - kudu-nio-2] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.703 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.720 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:22.721 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:22.721 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [1ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [1ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [15ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [15ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [32ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2124463346(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@6d05b5cd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@312808289 after retry RPC@1705808655), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3265d26d -> wakeup thread Time-limited test -> (continuation of Deferred@312808289 after retry RPC@1705808655))) after lookup
14:06:22.722 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:22.723 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.723 [DEBUG - kudu-nio-3] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.723 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.761 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:22.761 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:22.762 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=74), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [1ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [1ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [15ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [15ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [32ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [33ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [35ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [35ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [72ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2124463346(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@6d05b5cd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@312808289 after retry RPC@1705808655) -> (continuation of Deferred@1151758034 after retry RPC@1158920610), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3265d26d -> wakeup thread Time-limited test -> (continuation of Deferred@312808289 after retry RPC@1705808655) -> (continuation of Deferred@1151758034 after retry RPC@1158920610))) after lookup
14:06:22.763 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:22.764 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.764 [DEBUG - kudu-nio-4] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.764 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.779 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:22.780 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:22.780 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=92), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [1ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [1ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [15ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [15ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [32ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [33ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [35ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [35ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [72ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [74ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [76ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [76ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [91ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2124463346(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@6d05b5cd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@312808289 after retry RPC@1705808655) -> (continuation of Deferred@1151758034 after retry RPC@1158920610) -> (continuation of Deferred@43003166 after retry RPC@1511414343), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3265d26d -> wakeup thread Time-limited test -> (continuation of Deferred@312808289 after retry RPC@1705808655) -> (continuation of Deferred@1151758034 after retry RPC@1158920610) -> (continuation of Deferred@43003166 after retry RPC@1511414343))) after lookup
14:06:22.781 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:22.782 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.782 [DEBUG - kudu-nio-5] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:22.782 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:23.280 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:23.280 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:23.280 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=592), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [1ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [1ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [15ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [15ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [32ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [33ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [35ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [35ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [72ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [74ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [76ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [76ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [91ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [92ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [94ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [94ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [588ms] refreshing cache from master, [588ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [591ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2124463346(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@6d05b5cd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@312808289 after retry RPC@1705808655) -> (continuation of Deferred@1151758034 after retry RPC@1158920610) -> (continuation of Deferred@43003166 after retry RPC@1511414343) -> (continuation of Deferred@1488063801 after retry RPC@815702026), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3265d26d -> wakeup thread Time-limited test -> (continuation of Deferred@312808289 after retry RPC@1705808655) -> (continuation of Deferred@1151758034 after retry RPC@1158920610) -> (continuation of Deferred@43003166 after retry RPC@1511414343) -> (continuation of Deferred@1488063801 after retry RPC@815702026))) after lookup
14:06:23.281 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:23.282 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:23.283 [DEBUG - kudu-nio-6] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:23.283 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:23.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.668349 31774 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:06:23.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.875041 31787 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2c7b9c0f64b41c6b3f9a0e3084cf3e8)
14:06:23.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.875803 31787 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:23.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.877929 31787 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:23.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:23.882719 31463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111)
14:06:23.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:23.886559 31463 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127): Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111)
14:06:23.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.893801 31728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "957569291a7c402594d391a83985d059" is_pre_election: true
14:06:23.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.894743 31728 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1649b01489046e3a63ca563aad7869c in term 6.
14:06:23.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.896373 31463 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e1649b01489046e3a63ca563aad7869c; no voters: e2c7b9c0f64b41c6b3f9a0e3084cf3e8
14:06:23.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.897174 31787 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 7 FOLLOWER]: Leader pre-election won for term 8
14:06:23.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.897567 31787 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2c7b9c0f64b41c6b3f9a0e3084cf3e8)
14:06:23.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.898025 31787 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 7 FOLLOWER]: Advancing to term 8
14:06:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.905637 31787 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:23.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.907229 31787 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 8 election: Requested vote from peers e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:23.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.908795 31728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "957569291a7c402594d391a83985d059"
14:06:23.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.909241 31728 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 6 FOLLOWER]: Advancing to term 8
14:06:23.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:23.910580 31463 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127): Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111)
14:06:23.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.916113 31728 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1649b01489046e3a63ca563aad7869c in term 8.
14:06:23.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.917086 31463 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e1649b01489046e3a63ca563aad7869c; no voters: e2c7b9c0f64b41c6b3f9a0e3084cf3e8
14:06:23.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.917723 31787 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 8 FOLLOWER]: Leader election won for term 8
14:06:23.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.919109 31787 raft_consensus.cc:697] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 8 LEADER]: Becoming Leader. State: Replica: e1649b01489046e3a63ca563aad7869c, State: Running, Role: LEADER
14:06:23.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.920100 31787 consensus_queue.cc:237] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:23.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:23.929994 30276 catalog_manager.cc:5697] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c reported cstate change: term changed from 7 to 8, leader changed from e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195) to e1649b01489046e3a63ca563aad7869c (127.29.94.193). New cstate: current_term: 8 leader_uuid: "e1649b01489046e3a63ca563aad7869c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } health_report { overall_health: UNKNOWN } } }
14:06:24.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:24.350772 31463 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c -> Peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127): Couldn't send request to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:06:24.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:24.503772 31728 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 8 FOLLOWER]: Refusing update from remote peer e1649b01489046e3a63ca563aad7869c: Log matching property violated. Preceding OpId in replica: term: 6 index: 20. Preceding OpId from leader: term: 8 index: 25. (index mismatch)
14:06:24.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:24.505381 31787 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [LEADER]: Connected to new peer: Peer: permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
14:06:24.621 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:24.621 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:24.622 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1933), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [1ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [1ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [12ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [13ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [15ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [15ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [32ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [33ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [35ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [35ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [72ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [74ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [76ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [76ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [91ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [92ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [94ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [94ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [588ms] refreshing cache from master, [588ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [591ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [593ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [595ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [595ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [1928ms] refreshing cache from master, [1928ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [1932ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@2124463346(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@6d05b5cd -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@312808289 after retry RPC@1705808655) -> (continuation of Deferred@1151758034 after retry RPC@1158920610) -> (continuation of Deferred@43003166 after retry RPC@1511414343) -> (continuation of Deferred@1488063801 after retry RPC@815702026) -> (continuation of Deferred@46591787 after retry RPC@1193116041), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3265d26d -> wakeup thread Time-limited test -> (continuation of Deferred@312808289 after retry RPC@1705808655) -> (continuation of Deferred@1151758034 after retry RPC@1158920610) -> (continuation of Deferred@43003166 after retry RPC@1511414343) -> (continuation of Deferred@1488063801 after retry RPC@815702026) -> (continuation of Deferred@46591787 after retry RPC@1193116041))) after lookup
14:06:24.622 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:24.624 [DEBUG - kudu-nio-7] (Connection.java:757) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] Successfully connected to peer
14:06:24.653 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0xf3cee145, L:/127.0.0.1:38760 - R:/127.29.94.193:46741] using TOKEN/null
14:06:24.699 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.94.195:42127
14:06:24.700 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:06:24.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:24.701520 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:24.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/logs
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:24.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.195:42127
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.195
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35565
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.195
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:24.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:06:25.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:25.008673 31815 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:06:25.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:25.009234 31815 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:06:25.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:25.009804 31815 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:06:25.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:25.043009 31815 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:06:25.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:25.043985 31815 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.195
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:25.080577 31815 tablet_server_runner.cc:78] Tablet server non-default flags:
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.195:42127
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.195
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35565
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.195
14:06:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/logs
14:06:25.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:25.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:06:25.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:06:25.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:06:25.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:06:25.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:06:25.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:06:25.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:06:25.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:06:25.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:06:25.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:25.082252 31815 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:06:25.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:25.084069 31815 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:06:25.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:25.096230 31823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:26.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:25.097049 31824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:26.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:26.448544 31825 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1347 milliseconds
14:06:26.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:26.449471 31815 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.352s	user 0.492s	sys 0.830s
14:06:26.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:26.449858 31815 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.353s	user 0.492s	sys 0.830s
14:06:26.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:26.450037 31826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:26.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.450139 31815 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:06:26.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.451678 31815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:06:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.454308 31815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:06:26.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.455758 31815 hybrid_clock.cc:648] HybridClock initialized: now 1783519586455666 us; error 91 us; skew 500 ppm
14:06:26.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.456827 31815 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:06:26.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.465209 31815 webserver.cc:533] Webserver started at http://127.29.94.195:35565/ using document root <none> and password file <none>
14:06:26.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.466295 31815 fs_manager.cc:362] Metadata directory not provided
14:06:26.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.466554 31815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:06:26.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.475582 31815 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.005s	sys 0.002s
14:06:26.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.481623 31836 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:06:26.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.482975 31815 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.003s
14:06:26.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.483403 31815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:26.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:06:26.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:54 on dist-test-slave-59nb"
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.485479 31815 fs_report.cc:389] FS layout report
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/data
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:06:26.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:06:26.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.573444 31815 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:06:26.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.575344 31815 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:06:26.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.575786 31815 kserver.cc:163] Server-wide thread pool size limit: 3276
14:06:26.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.578778 31815 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:06:26.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.585124 31843 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:06:26.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.593178 31815 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:06:26.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.593417 31815 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
14:06:26.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.593653 31815 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:06:26.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.599002 31815 ts_tablet_manager.cc:616] Registered 1 tablets
14:06:26.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.599206 31815 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.006s	sys 0.000s
14:06:26.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.599773 31843 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Bootstrap starting.
14:06:26.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:26.659183 31770 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf420 after lost signal to thread 31637
14:06:26.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:26.660378 31770 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfe20 after lost signal to thread 31773
14:06:26.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.681975 31843 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Log is configured to *not* fsync() on all Append() calls
14:06:26.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.792095 31843 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:06:26.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.793114 31843 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Bootstrap complete.
14:06:26.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.794823 31843 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Time spent bootstrapping tablet: real 0.196s	user 0.123s	sys 0.058s
14:06:26.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.820703 31843 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:26.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.823791 31843 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c7b9c0f64b41c6b3f9a0e3084cf3e8, State: Initialized, Role: FOLLOWER
14:06:26.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.824843 31843 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:26.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.830358 31843 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Time spent starting tablet: real 0.035s	user 0.028s	sys 0.002s
14:06:26.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.834098 31815 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.195:42127
14:06:26.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.834543 31951 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.195:42127 every 8 connection(s)
14:06:26.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.837071 31815 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:06:26.840 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:26.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.838384 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 31815
14:06:26.918 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:06:26.918 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.94.193:46741
14:06:26.919 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:06:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.920064 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 31438
14:06:26.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.946895 31959 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:06:26.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.947489 31959 heartbeater.cc:461] Registering TS with master...
14:06:26.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.948936 31959 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:06:26.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.952776 31952 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:06:26.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.953199 31952 heartbeater.cc:461] Registering TS with master...
14:06:26.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.954186 31952 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:06:26.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.954650 31953 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:06:26.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.955049 31953 heartbeater.cc:461] Registering TS with master...
14:06:26.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.956125 31953 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:06:26.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.956357 30120 ts_manager.cc:194] Re-registered known tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:26.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.964927 30276 ts_manager.cc:194] Re-registered known tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:26.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.967993 30276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.195:39619
14:06:26.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:26.969429 30195 ts_manager.cc:194] Re-registered known tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:26.992 [DEBUG - kudu-nio-7] (Connection.java:239) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] handling channelInactive
14:06:26.993 [DEBUG - kudu-nio-7] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state READY due to: connection closed
14:06:26.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:26.992859 31963 negotiation.cc:336] Failed RPC negotiation. Trace:
14:06:26.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:26.887195 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.29.94.193:55393 (local address 127.29.94.195:42127)
14:06:26.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:26.927042 (+ 39847us) server_negotiation.cc:207] Beginning negotiation
14:06:26.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:26.927072 (+    30us) server_negotiation.cc:400] Waiting for connection header
14:06:26.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:26.927143 (+    71us) server_negotiation.cc:408] Connection header received
14:06:26.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:26.927353 (+   210us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
14:06:26.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:26.927385 (+    32us) server_negotiation.cc:462] Received NEGOTIATE request from client
14:06:26.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:26.927766 (+   381us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
14:06:26.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:26.927992 (+   226us) server_negotiation.cc:227] Negotiated authn=SASL
14:06:26.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:26.990946 (+ 62954us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.94.193:55393: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
14:06:26.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":30693,"thread_start_us":37905,"threads_started":1}
14:06:26.997 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:26.998 [DEBUG - Time-limited test] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:26.999 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:26.999 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:26.999 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.022 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:27.022 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:27.023 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@765857167(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@35dcad6c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48e303ea -> wakeup thread Time-limited test)) after lookup
14:06:27.025 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:27.027 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.027 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.027 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.040 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:27.041 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:27.041 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [28ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [30ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [30ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [43ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@765857167(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@35dcad6c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48e303ea -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348))) after lookup
14:06:27.042 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:27.044 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.044 [DEBUG - kudu-nio-2] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.044 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.081 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:27.082 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:27.082 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [28ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [30ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [30ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [43ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [45ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [47ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [47ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [79ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@765857167(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@35dcad6c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348) -> (continuation of Deferred@1658794733 after retry RPC@1077901233), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48e303ea -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348) -> (continuation of Deferred@1658794733 after retry RPC@1077901233))) after lookup
14:06:27.083 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:27.084 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.084 [DEBUG - kudu-nio-3] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.084 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.141 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:27.141 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:27.142 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=144), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [28ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [30ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [30ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [43ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [45ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [47ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [47ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [79ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [85ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [87ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [87ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [138ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [143ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@765857167(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@35dcad6c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348) -> (continuation of Deferred@1658794733 after retry RPC@1077901233) -> (continuation of Deferred@1264054830 after retry RPC@1886291466), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48e303ea -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348) -> (continuation of Deferred@1658794733 after retry RPC@1077901233) -> (continuation of Deferred@1264054830 after retry RPC@1886291466))) after lookup
14:06:27.143 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:27.144 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.144 [DEBUG - kudu-nio-4] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.144 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.461 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:27.461 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:27.463 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=465), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [28ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [30ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [30ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [43ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [45ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [47ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [47ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [79ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [85ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [87ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [87ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [138ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [143ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [145ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [147ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [148ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [459ms] refreshing cache from master, [459ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [464ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@765857167(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@35dcad6c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348) -> (continuation of Deferred@1658794733 after retry RPC@1077901233) -> (continuation of Deferred@1264054830 after retry RPC@1886291466) -> (continuation of Deferred@1141364539 after retry RPC@1088247655), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48e303ea -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348) -> (continuation of Deferred@1658794733 after retry RPC@1077901233) -> (continuation of Deferred@1264054830 after retry RPC@1886291466) -> (continuation of Deferred@1141364539 after retry RPC@1088247655))) after lookup
14:06:27.463 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:27.465 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.465 [DEBUG - kudu-nio-5] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.466 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.563 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:27.563 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:27.564 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=566), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [28ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [30ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [30ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [43ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [45ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [47ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [47ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [79ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [85ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [87ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [87ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [138ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [143ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [145ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [147ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [148ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [459ms] refreshing cache from master, [459ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [464ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [466ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [468ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [469ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [560ms] refreshing cache from master, [560ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [565ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@765857167(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@35dcad6c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348) -> (continuation of Deferred@1658794733 after retry RPC@1077901233) -> (continuation of Deferred@1264054830 after retry RPC@1886291466) -> (continuation of Deferred@1141364539 after retry RPC@1088247655) -> (continuation of Deferred@1914744483 after retry RPC@1483516399), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48e303ea -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348) -> (continuation of Deferred@1658794733 after retry RPC@1077901233) -> (continuation of Deferred@1264054830 after retry RPC@1886291466) -> (continuation of Deferred@1141364539 after retry RPC@1088247655) -> (continuation of Deferred@1914744483 after retry RPC@1483516399))) after lookup
14:06:27.565 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:27.567 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.570 [DEBUG - kudu-nio-6] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.571 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:27.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:27.972775 31952 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:06:28.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.357010 31965 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e1649b01489046e3a63ca563aad7869c)
14:06:28.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.357558 31965 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:28.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.360602 31965 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:28.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:28.365585 31662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:06:28.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:28.371927 31662 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:06:28.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.375341 31906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" is_pre_election: true
14:06:28.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.376324 31906 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957569291a7c402594d391a83985d059 in term 7.
14:06:28.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.377943 31662 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: e1649b01489046e3a63ca563aad7869c
14:06:28.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.378818 31965 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 8 FOLLOWER]: Leader pre-election won for term 9
14:06:28.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.379096 31965 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e1649b01489046e3a63ca563aad7869c)
14:06:28.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.379321 31965 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 8 FOLLOWER]: Advancing to term 9
14:06:28.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.383725 31965 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:28.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.385133 31965 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 9 election: Requested vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:28.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.386881 31906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:06:28.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.387437 31906 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 7 FOLLOWER]: Advancing to term 9
14:06:28.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:28.388964 31662 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:06:28.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.393270 31906 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957569291a7c402594d391a83985d059 in term 9.
14:06:28.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.394243 31662 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: e1649b01489046e3a63ca563aad7869c
14:06:28.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.395153 31965 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 9 FOLLOWER]: Leader election won for term 9
14:06:28.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.396965 31965 raft_consensus.cc:697] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 9 LEADER]: Becoming Leader. State: Replica: 957569291a7c402594d391a83985d059, State: Running, Role: LEADER
14:06:28.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.397913 31965 consensus_queue.cc:237] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:28.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.408999 30276 catalog_manager.cc:5697] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 reported cstate change: term changed from 8 to 9, leader changed from e1649b01489046e3a63ca563aad7869c (127.29.94.193) to 957569291a7c402594d391a83985d059 (127.29.94.194). New cstate: current_term: 9 leader_uuid: "957569291a7c402594d391a83985d059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } health_report { overall_health: HEALTHY } } }
14:06:28.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:28.791141 31662 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 -> Peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Couldn't send request to peer e1649b01489046e3a63ca563aad7869c. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:06:28.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.809770 31906 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 9 FOLLOWER]: Refusing update from remote peer 957569291a7c402594d391a83985d059: Log matching property violated. Preceding OpId in replica: term: 7 index: 24. Preceding OpId from leader: term: 9 index: 29. (index mismatch)
14:06:28.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:28.811189 31965 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
14:06:30.200 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:30.201 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:30.201 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=3204), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [24ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [28ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [30ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [30ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [43ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [45ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [47ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [47ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [79ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [84ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [85ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [87ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [87ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [138ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [143ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [145ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [147ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [148ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [459ms] refreshing cache from master, [459ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [464ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [466ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [468ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [469ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [560ms] refreshing cache from master, [560ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [565ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [567ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [573ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [574ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [3199ms] refreshing cache from master, [3199ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [3203ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@765857167(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@35dcad6c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348) -> (continuation of Deferred@1658794733 after retry RPC@1077901233) -> (continuation of Deferred@1264054830 after retry RPC@1886291466) -> (continuation of Deferred@1141364539 after retry RPC@1088247655) -> (continuation of Deferred@1914744483 after retry RPC@1483516399) -> (continuation of Deferred@466610150 after retry RPC@651237025), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48e303ea -> wakeup thread Time-limited test -> (continuation of Deferred@95569027 after retry RPC@1436478348) -> (continuation of Deferred@1658794733 after retry RPC@1077901233) -> (continuation of Deferred@1264054830 after retry RPC@1886291466) -> (continuation of Deferred@1141364539 after retry RPC@1088247655) -> (continuation of Deferred@1914744483 after retry RPC@1483516399) -> (continuation of Deferred@466610150 after retry RPC@651237025))) after lookup
14:06:30.202 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:30.204 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] Successfully connected to peer
14:06:30.225 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x7615d59b, L:/127.0.0.1:35734 - R:/127.29.94.194:43301] using TOKEN/null
14:06:30.273 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.94.193:46741
14:06:30.273 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:30.274789 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/logs
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.193:46741
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.193
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35541
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.193
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:30.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:06:30.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:30.613655 32003 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:06:30.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:30.614212 32003 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:06:30.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:30.614701 32003 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:06:30.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:30.680236 32003 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:06:30.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:30.681409 32003 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.193
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:30.723528 32003 tablet_server_runner.cc:78] Tablet server non-default flags:
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.193:46741
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.193
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35541
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:06:30.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.193
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/logs
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:06:30.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:30.725522 32003 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:06:30.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:30.727577 32003 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:06:30.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:30.742750 32011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:31.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:31.490960 31662 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 -> Peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Couldn't send request to peer e1649b01489046e3a63ca563aad7869c. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:06:31.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:31.983623 32014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:31.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:30.747769 32012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:31.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:31.988480 32013 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1236 milliseconds
14:06:31.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:31.988639 32003 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:06:31.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:31.990087 32003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:06:31.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:31.992580 32003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:06:31.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:31.993978 32003 hybrid_clock.cc:648] HybridClock initialized: now 1783519591993921 us; error 60 us; skew 500 ppm
14:06:31.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:31.994786 32003 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:06:32.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.000985 32003 webserver.cc:533] Webserver started at http://127.29.94.193:35541/ using document root <none> and password file <none>
14:06:32.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.001845 32003 fs_manager.cc:362] Metadata directory not provided
14:06:32.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.002032 32003 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:06:32.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.010244 32003 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.000s
14:06:32.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.015192 32026 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:06:32.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.016680 32003 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.004s
14:06:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.017149 32003 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e1649b01489046e3a63ca563aad7869c"
14:06:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:51 on dist-test-slave-59nb"
14:06:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.020057 32003 fs_report.cc:389] FS layout report
14:06:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:06:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/data
14:06:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:06:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:06:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:06:32.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:06:32.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:06:32.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:06:32.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:06:32.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:06:32.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:06:32.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:06:32.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:06:32.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.084187 32003 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:06:32.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.085846 32003 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:06:32.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.086293 32003 kserver.cc:163] Server-wide thread pool size limit: 3276
14:06:32.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.089272 32003 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:06:32.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.095567 32033 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:06:32.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.104164 32003 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:06:32.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.104447 32003 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
14:06:32.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.104773 32003 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:06:32.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.110563 32003 ts_tablet_manager.cc:616] Registered 1 tablets
14:06:32.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.110759 32003 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.000s	sys 0.003s
14:06:32.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.111311 32033 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap starting.
14:06:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.186676 32033 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Log is configured to *not* fsync() on all Append() calls
14:06:32.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.288480 32003 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.193:46741
14:06:32.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.288594 32140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.193:46741 every 8 connection(s)
14:06:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.292676 32003 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:32.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.295954 32033 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:06:32.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.297025 32033 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap complete.
14:06:32.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.298867 32033 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Time spent bootstrapping tablet: real 0.188s	user 0.159s	sys 0.026s
14:06:32.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.300194 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 32003
14:06:32.301 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:32.337 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:06:32.339 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.94.194:43301
14:06:32.339 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:06:32.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.339960 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 31636
14:06:32.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.338492 32033 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:32.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.342907 32033 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1649b01489046e3a63ca563aad7869c, State: Initialized, Role: FOLLOWER
14:06:32.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.344879 32033 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:32.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.364084 32033 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Time spent starting tablet: real 0.065s	user 0.031s	sys 0.004s
14:06:32.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.365166 32141 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:06:32.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.365697 32141 heartbeater.cc:461] Registering TS with master...
14:06:32.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.367033 32141 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:06:32.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.371027 32142 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:06:32.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.371373 32142 heartbeater.cc:461] Registering TS with master...
14:06:32.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.370725 30276 ts_manager.cc:194] Re-registered known tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:06:32.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.372184 32142 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:06:32.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.375217 30276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.193:55311
14:06:32.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.375586 30196 ts_manager.cc:194] Re-registered known tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:06:32.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.377381 32143 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:06:32.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.377724 32143 heartbeater.cc:461] Registering TS with master...
14:06:32.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.378643 32143 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:06:32.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.381079 32141 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:06:32.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:32.382344 30120 ts_manager.cc:194] Re-registered known tserver with Master: e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741)
14:06:32.389 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] handling channelInactive
14:06:32.389 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:32.389 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state READY due to: connection closed
14:06:32.390 [DEBUG - Time-limited test] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:32.391 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.391 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.391 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.400 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:32.400 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:32.400 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [1ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1369776861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@142fc00e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14010d85 -> wakeup thread Time-limited test)) after lookup
14:06:32.401 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:32.402 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.402 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.402 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.420 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:32.420 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:32.421 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [1ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [11ms] sending RPC to server 957569291a7c402594d391a83985d059, [12ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [12ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [30ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1369776861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@142fc00e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14010d85 -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994))) after lookup
14:06:32.421 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:32.423 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.423 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.423 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.461 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:32.461 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:32.463 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [1ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [11ms] sending RPC to server 957569291a7c402594d391a83985d059, [12ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [12ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [30ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [31ms] sending RPC to server 957569291a7c402594d391a83985d059, [33ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [33ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [70ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1369776861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@142fc00e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994) -> (continuation of Deferred@1683475941 after retry RPC@1384809587), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14010d85 -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994) -> (continuation of Deferred@1683475941 after retry RPC@1384809587))) after lookup
14:06:32.463 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:32.465 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.465 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.465 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.520 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:32.521 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:32.521 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=131), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [1ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [11ms] sending RPC to server 957569291a7c402594d391a83985d059, [12ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [12ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [30ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [31ms] sending RPC to server 957569291a7c402594d391a83985d059, [33ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [33ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [70ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [73ms] sending RPC to server 957569291a7c402594d391a83985d059, [75ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [75ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [126ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [130ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1369776861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@142fc00e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994) -> (continuation of Deferred@1683475941 after retry RPC@1384809587) -> (continuation of Deferred@886357457 after retry RPC@846929444), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14010d85 -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994) -> (continuation of Deferred@1683475941 after retry RPC@1384809587) -> (continuation of Deferred@886357457 after retry RPC@846929444))) after lookup
14:06:32.522 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:32.523 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.523 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.523 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.540 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:32.540 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:32.540 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=150), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [1ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [11ms] sending RPC to server 957569291a7c402594d391a83985d059, [12ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [12ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [30ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [31ms] sending RPC to server 957569291a7c402594d391a83985d059, [33ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [33ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [70ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [73ms] sending RPC to server 957569291a7c402594d391a83985d059, [75ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [75ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [126ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [130ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [131ms] sending RPC to server 957569291a7c402594d391a83985d059, [133ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [133ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [149ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1369776861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@142fc00e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994) -> (continuation of Deferred@1683475941 after retry RPC@1384809587) -> (continuation of Deferred@886357457 after retry RPC@846929444) -> (continuation of Deferred@420259644 after retry RPC@1944850439), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14010d85 -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994) -> (continuation of Deferred@1683475941 after retry RPC@1384809587) -> (continuation of Deferred@886357457 after retry RPC@846929444) -> (continuation of Deferred@420259644 after retry RPC@1944850439))) after lookup
14:06:32.541 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:32.542 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.542 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.542 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.760 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:32.760 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:32.761 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=371), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [1ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [11ms] sending RPC to server 957569291a7c402594d391a83985d059, [12ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [12ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [30ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [31ms] sending RPC to server 957569291a7c402594d391a83985d059, [33ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [33ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [70ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [73ms] sending RPC to server 957569291a7c402594d391a83985d059, [75ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [75ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [126ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [130ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [131ms] sending RPC to server 957569291a7c402594d391a83985d059, [133ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [133ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [149ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [151ms] sending RPC to server 957569291a7c402594d391a83985d059, [152ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [152ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [366ms] refreshing cache from master, [366ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [370ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1369776861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@142fc00e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994) -> (continuation of Deferred@1683475941 after retry RPC@1384809587) -> (continuation of Deferred@886357457 after retry RPC@846929444) -> (continuation of Deferred@420259644 after retry RPC@1944850439) -> (continuation of Deferred@1248642925 after retry RPC@1494452751), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14010d85 -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994) -> (continuation of Deferred@1683475941 after retry RPC@1384809587) -> (continuation of Deferred@886357457 after retry RPC@846929444) -> (continuation of Deferred@420259644 after retry RPC@1944850439) -> (continuation of Deferred@1248642925 after retry RPC@1494452751))) after lookup
14:06:32.761 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:32.763 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.763 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state CONNECTING due to: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:32.763 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 957569291a7c402594d391a83985d059(127.29.94.194:43301) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301
14:06:33.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.544984 32155 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:06:33.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.545568 32155 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:33.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.548436 32155 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:33.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:33.554540 32030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:33.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:33.559451 32030 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:33.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.560482 31906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" is_pre_election: true
14:06:33.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.562074 32030 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1649b01489046e3a63ca563aad7869c; no voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8
14:06:33.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.562841 32155 raft_consensus.cc:2749] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
14:06:33.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.681977 32160 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 957569291a7c402594d391a83985d059)
14:06:33.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.682626 32160 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:33.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.686285 32160 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:33.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:33.690527 31840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:33.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:33.694623 31840 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:33.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.697956 32096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "e1649b01489046e3a63ca563aad7869c" is_pre_election: true
14:06:33.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.698632 32096 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c7b9c0f64b41c6b3f9a0e3084cf3e8 in term 8.
14:06:33.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.700098 31840 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e1649b01489046e3a63ca563aad7869c, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: 957569291a7c402594d391a83985d059
14:06:33.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.700917 32160 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 9 FOLLOWER]: Leader pre-election won for term 10
14:06:33.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.701182 32160 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 957569291a7c402594d391a83985d059)
14:06:33.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.701416 32160 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 9 FOLLOWER]: Advancing to term 10
14:06:33.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.705976 32160 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:33.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.707486 32160 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 10 election: Requested vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:33.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.708355 32096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "e1649b01489046e3a63ca563aad7869c"
14:06:33.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.708782 32096 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 8 FOLLOWER]: Advancing to term 10
14:06:33.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:33.711252 31840 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111)
14:06:33.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.715130 32096 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c7b9c0f64b41c6b3f9a0e3084cf3e8 in term 10.
14:06:33.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.716117 31840 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e1649b01489046e3a63ca563aad7869c, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: 957569291a7c402594d391a83985d059
14:06:33.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.716840 32160 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 10 FOLLOWER]: Leader election won for term 10
14:06:33.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.718406 32160 raft_consensus.cc:697] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 10 LEADER]: Becoming Leader. State: Replica: e2c7b9c0f64b41c6b3f9a0e3084cf3e8, State: Running, Role: LEADER
14:06:33.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.719386 32160 consensus_queue.cc:237] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:33.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.728860 30276 catalog_manager.cc:5697] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 reported cstate change: term changed from 9 to 10, leader changed from 957569291a7c402594d391a83985d059 (127.29.94.194) to e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195). New cstate: current_term: 10 leader_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } health_report { overall_health: UNKNOWN } } }
14:06:33.881 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:33.881 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:33.881 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=14, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1491), Traces: [0ms] sending RPC to server 957569291a7c402594d391a83985d059, [1ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [2ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [10ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [11ms] sending RPC to server 957569291a7c402594d391a83985d059, [12ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [12ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [30ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [31ms] sending RPC to server 957569291a7c402594d391a83985d059, [33ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [33ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [70ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [73ms] sending RPC to server 957569291a7c402594d391a83985d059, [75ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [75ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [126ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [130ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [131ms] sending RPC to server 957569291a7c402594d391a83985d059, [133ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [133ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [149ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [151ms] sending RPC to server 957569291a7c402594d391a83985d059, [152ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [152ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [366ms] refreshing cache from master, [366ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [370ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [371ms] sending RPC to server 957569291a7c402594d391a83985d059, [373ms] received response from server 957569291a7c402594d391a83985d059: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [373ms] delaying RPC due to: Network error: Failed to connect to peer 957569291a7c402594d391a83985d059(127.29.94.194:43301): Connection refused: /127.29.94.194:43301, [1486ms] refreshing cache from master, [1486ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [1490ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1369776861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@142fc00e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994) -> (continuation of Deferred@1683475941 after retry RPC@1384809587) -> (continuation of Deferred@886357457 after retry RPC@846929444) -> (continuation of Deferred@420259644 after retry RPC@1944850439) -> (continuation of Deferred@1248642925 after retry RPC@1494452751) -> (continuation of Deferred@568840593 after retry RPC@1684108368), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@14010d85 -> wakeup thread Time-limited test -> (continuation of Deferred@2079029759 after retry RPC@1729768994) -> (continuation of Deferred@1683475941 after retry RPC@1384809587) -> (continuation of Deferred@886357457 after retry RPC@846929444) -> (continuation of Deferred@420259644 after retry RPC@1944850439) -> (continuation of Deferred@1248642925 after retry RPC@1494452751) -> (continuation of Deferred@568840593 after retry RPC@1684108368))) after lookup
14:06:33.882 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:33.883 [DEBUG - kudu-nio-7] (Connection.java:757) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] Successfully connected to peer
14:06:33.902 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x2ec2f6eb, L:/127.0.0.1:60208 - R:/127.29.94.195:42127] using TOKEN/null
14:06:33.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.912845 32096 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 10 FOLLOWER]: Refusing update from remote peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Log matching property violated. Preceding OpId in replica: term: 8 index: 28. Preceding OpId from leader: term: 10 index: 34. (index mismatch)
14:06:33.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:33.913825 31840 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 -> Peer 957569291a7c402594d391a83985d059 (127.29.94.194:43301): Couldn't send request to peer 957569291a7c402594d391a83985d059. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.194:43301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:06:33.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.914943 32165 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.001s
14:06:33.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:33.942667 32179 mvcc.cc:204] Tried to move back new op lower bound from 7305296256648216576 to 7305296255883001856. Current Snapshot: MvccSnapshot[applied={T|T < 7305296241715056640}]
14:06:34.017 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.94.194:43301
14:06:34.017 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:06:34.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:34.019008 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:34.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:34.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:34.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data
14:06:34.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:06:34.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:06:34.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:06:34.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/logs
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.194:43301
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.194
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33225
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.194
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:06:34.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:34.383312 32185 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:06:34.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:34.383795 32185 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:06:34.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:34.384238 32185 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:06:34.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:34.417083 32185 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:06:34.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:34.417920 32185 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.194
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:34.455387 32185 tablet_server_runner.cc:78] Tablet server non-default flags:
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.194:43301
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.194
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33225
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:06:34.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.194
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/logs
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:06:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:34.456851 32185 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:06:34.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:34.458491 32185 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:06:34.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:34.472323 32191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:35.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:35.050011 31947 debug-util.cc:398] Leaking SignalData structure 0x7b08000be8a0 after lost signal to thread 31818
14:06:35.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:35.051738 31947 debug-util.cc:398] Leaking SignalData structure 0x7b08000be860 after lost signal to thread 31951
14:06:35.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:35.658465 30227 debug-util.cc:398] Leaking SignalData structure 0x7b0800098840 after lost signal to thread 30162
14:06:35.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:35.659330 30227 debug-util.cc:398] Leaking SignalData structure 0x7b0800070140 after lost signal to thread 30230
14:06:35.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:34.476505 32194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:35.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:34.473379 32192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:35.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:35.704159 32193 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1221 milliseconds
14:06:35.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.704291 32185 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:06:35.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.706084 32185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:06:35.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.712146 32185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:06:35.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.713696 32185 hybrid_clock.cc:648] HybridClock initialized: now 1783519595713621 us; error 101 us; skew 500 ppm
14:06:35.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.714660 32185 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:06:35.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.723059 32185 webserver.cc:533] Webserver started at http://127.29.94.194:33225/ using document root <none> and password file <none>
14:06:35.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.724141 32185 fs_manager.cc:362] Metadata directory not provided
14:06:35.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.724385 32185 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:06:35.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.733455 32185 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.001s	sys 0.004s
14:06:35.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.738879 32207 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:06:35.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.740108 32185 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
14:06:35.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.740458 32185 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "957569291a7c402594d391a83985d059"
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:53 on dist-test-slave-59nb"
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.742590 32185 fs_report.cc:389] FS layout report
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/wal
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/data
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:06:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:06:35.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.808153 32185 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:06:35.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.809890 32185 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:06:35.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.810393 32185 kserver.cc:163] Server-wide thread pool size limit: 3276
14:06:35.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.813856 32185 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:06:35.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.821190 32214 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:06:35.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.830065 32185 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:06:35.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.830302 32185 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.000s
14:06:35.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.830724 32185 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:06:35.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.836371 32185 ts_tablet_manager.cc:616] Registered 1 tablets
14:06:35.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.836614 32185 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
14:06:35.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.837934 32214 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Bootstrap starting.
14:06:35.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:35.972517 32214 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Log is configured to *not* fsync() on all Append() calls
14:06:36.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.120193 32214 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:06:36.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.121176 32214 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Bootstrap complete.
14:06:36.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.122853 32214 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Time spent bootstrapping tablet: real 0.286s	user 0.168s	sys 0.040s
14:06:36.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.144614 32214 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:36.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.147634 32214 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957569291a7c402594d391a83985d059, State: Initialized, Role: FOLLOWER
14:06:36.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.148874 32214 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:36.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.159552 32214 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059: Time spent starting tablet: real 0.036s	user 0.022s	sys 0.012s
14:06:36.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.366434 32185 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.194:43301
14:06:36.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.367110 32322 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.194:43301 every 8 connection(s)
14:06:36.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.369894 32185 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-1/data/info.pb
14:06:36.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.372992 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 32185
14:06:36.373 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:36.455 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:06:36.455 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.94.195:42127
14:06:36.456 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:06:36.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.456759 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 31815
14:06:36.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.466897 32325 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:06:36.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.467428 32325 heartbeater.cc:461] Registering TS with master...
14:06:36.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.468779 32325 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:06:36.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.470431 32323 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:06:36.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.470775 32323 heartbeater.cc:461] Registering TS with master...
14:06:36.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.471745 32323 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:06:36.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.474426 30196 ts_manager.cc:194] Re-registered known tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:36.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.477871 30276 ts_manager.cc:194] Re-registered known tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:36.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.481065 30276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.194:49089
14:06:36.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.482316 32328 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:06:36.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.482647 32328 heartbeater.cc:461] Registering TS with master...
14:06:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.483526 32328 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:06:36.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:36.487123 30121 ts_manager.cc:194] Re-registered known tserver with Master: 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:36.505 [DEBUG - kudu-nio-7] (Connection.java:239) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] handling channelInactive
14:06:36.505 [DEBUG - kudu-nio-7] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state READY due to: connection closed
14:06:36.506 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:36.506 [DEBUG - Time-limited test] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:36.508 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.508 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.508 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.520 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:36.521 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:36.521 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1514020439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@5c3e9abe -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6419f2db -> wakeup thread Time-limited test)) after lookup
14:06:36.521 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:36.523 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.523 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.523 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.540 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:36.540 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:36.541 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [15ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [17ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [17ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1514020439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@5c3e9abe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6419f2db -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670))) after lookup
14:06:36.541 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:36.543 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.543 [DEBUG - kudu-nio-2] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.543 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.587 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:36.589 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299998, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:36.589 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [15ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [17ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [17ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [35ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [37ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [37ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1514020439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@5c3e9abe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670) -> (continuation of Deferred@900035546 after retry RPC@1827379314), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6419f2db -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670) -> (continuation of Deferred@900035546 after retry RPC@1827379314))) after lookup
14:06:36.590 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:36.592 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.592 [DEBUG - kudu-nio-3] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.592 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.681 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:36.681 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:36.682 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=175), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [15ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [17ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [17ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [35ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [37ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [37ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [84ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [86ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [86ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [170ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [175ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1514020439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@5c3e9abe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670) -> (continuation of Deferred@900035546 after retry RPC@1827379314) -> (continuation of Deferred@1127705915 after retry RPC@1940870439), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6419f2db -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670) -> (continuation of Deferred@900035546 after retry RPC@1827379314) -> (continuation of Deferred@1127705915 after retry RPC@1940870439))) after lookup
14:06:36.683 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:36.685 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.685 [DEBUG - kudu-nio-4] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.685 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.961 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:36.961 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:36.962 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=455), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [15ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [17ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [17ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [35ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [37ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [37ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [84ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [86ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [86ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [170ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [175ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [177ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [179ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [179ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [450ms] refreshing cache from master, [451ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [454ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1514020439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@5c3e9abe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670) -> (continuation of Deferred@900035546 after retry RPC@1827379314) -> (continuation of Deferred@1127705915 after retry RPC@1940870439) -> (continuation of Deferred@1370158249 after retry RPC@1487246806), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6419f2db -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670) -> (continuation of Deferred@900035546 after retry RPC@1827379314) -> (continuation of Deferred@1127705915 after retry RPC@1940870439) -> (continuation of Deferred@1370158249 after retry RPC@1487246806))) after lookup
14:06:36.962 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:36.963 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.964 [DEBUG - kudu-nio-5] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:36.964 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:37.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.305053 32334 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:06:37.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.305593 32334 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:37.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.308032 32334 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:37.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:37.314132 32211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111)
14:06:37.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:37.318571 32211 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127): Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111)
14:06:37.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.323300 32096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "e1649b01489046e3a63ca563aad7869c" is_pre_election: true
14:06:37.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.324615 32211 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 957569291a7c402594d391a83985d059; no voters: e1649b01489046e3a63ca563aad7869c, e2c7b9c0f64b41c6b3f9a0e3084cf3e8
14:06:37.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.325273 32334 raft_consensus.cc:2749] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
14:06:37.481 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:37.482 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)[L]]}]
14:06:37.483 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=977), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [15ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [17ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [17ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [35ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [37ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [37ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [84ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [86ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [86ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [170ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [175ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [177ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [179ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [179ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [450ms] refreshing cache from master, [451ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [454ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [456ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [458ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [458ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [970ms] refreshing cache from master, [971ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [975ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1514020439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@5c3e9abe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670) -> (continuation of Deferred@900035546 after retry RPC@1827379314) -> (continuation of Deferred@1127705915 after retry RPC@1940870439) -> (continuation of Deferred@1370158249 after retry RPC@1487246806) -> (continuation of Deferred@1754163160 after retry RPC@485573690), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6419f2db -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670) -> (continuation of Deferred@900035546 after retry RPC@1827379314) -> (continuation of Deferred@1127705915 after retry RPC@1940870439) -> (continuation of Deferred@1370158249 after retry RPC@1487246806) -> (continuation of Deferred@1754163160 after retry RPC@485573690))) after lookup
14:06:37.484 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] connecting to peer
14:06:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.485478 32323 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:06:37.486 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:37.486 [DEBUG - kudu-nio-6] (Connection.java:709) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] cleaning up while in state CONNECTING due to: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:37.486 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127
14:06:37.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:37.614705 30152 debug-util.cc:398] Leaking SignalData structure 0x7b0800089380 after lost signal to thread 30089
14:06:37.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:37.615432 30152 debug-util.cc:398] Leaking SignalData structure 0x7b0800088640 after lost signal to thread 30155
14:06:37.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.767544 32339 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e2c7b9c0f64b41c6b3f9a0e3084cf3e8)
14:06:37.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.768026 32339 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:37.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.769829 32339 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:37.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:37.778789 32030 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127): Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111)
14:06:37.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.784472 32278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "957569291a7c402594d391a83985d059" is_pre_election: true
14:06:37.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.785218 32278 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1649b01489046e3a63ca563aad7869c in term 9.
14:06:37.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.786495 32030 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e1649b01489046e3a63ca563aad7869c; no voters: e2c7b9c0f64b41c6b3f9a0e3084cf3e8
14:06:37.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.787091 32339 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 10 FOLLOWER]: Leader pre-election won for term 11
14:06:37.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.787388 32339 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 10 FOLLOWER]: Starting leader election (detected failure of leader e2c7b9c0f64b41c6b3f9a0e3084cf3e8)
14:06:37.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.787616 32339 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 10 FOLLOWER]: Advancing to term 11
14:06:37.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.791864 32339 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:37.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.793388 32339 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 11 election: Requested vote from peers e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127), 957569291a7c402594d391a83985d059 (127.29.94.194:43301)
14:06:37.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.794739 32278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "e1649b01489046e3a63ca563aad7869c" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "957569291a7c402594d391a83985d059"
14:06:37.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.795279 32278 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 9 FOLLOWER]: Advancing to term 11
14:06:37.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:37.797389 32030 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127): Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111)
14:06:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.801745 32278 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1649b01489046e3a63ca563aad7869c in term 11.
14:06:37.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.802549 32030 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e1649b01489046e3a63ca563aad7869c; no voters: e2c7b9c0f64b41c6b3f9a0e3084cf3e8
14:06:37.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.803133 32339 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 11 FOLLOWER]: Leader election won for term 11
14:06:37.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.804348 32339 raft_consensus.cc:697] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 11 LEADER]: Becoming Leader. State: Replica: e1649b01489046e3a63ca563aad7869c, State: Running, Role: LEADER
14:06:37.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.805238 32339 consensus_queue.cc:237] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:37.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:37.815265 30276 catalog_manager.cc:5697] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c reported cstate change: term changed from 10 to 11, leader changed from e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195) to e1649b01489046e3a63ca563aad7869c (127.29.94.193). New cstate: current_term: 11 leader_uuid: "e1649b01489046e3a63ca563aad7869c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } health_report { overall_health: UNKNOWN } } }
14:06:38.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:38.229786 32278 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 11 FOLLOWER]: Refusing update from remote peer e1649b01489046e3a63ca563aad7869c: Log matching property violated. Preceding OpId in replica: term: 9 index: 32. Preceding OpId from leader: term: 11 index: 37. (index mismatch)
14:06:38.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:38.231115 32339 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [LEADER]: Connected to new peer: Peer: permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
14:06:38.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:38.274783 32030 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c -> Peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127): Couldn't send request to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:06:38.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:38.647732 30227 debug-util.cc:398] Leaking SignalData structure 0x7b08000a8300 after lost signal to thread 30162
14:06:38.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:38.648389 30227 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2a80 after lost signal to thread 30230
14:06:38.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:38.856153 32030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111) [suppressed 3 similar messages]
14:06:39.001 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:39.002 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:39.003 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=2496), Traces: [0ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [2ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [3ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [15ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [17ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [17ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [33ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [35ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [37ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [37ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [81ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [84ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [86ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [86ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [170ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [175ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [177ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [179ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [179ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [450ms] refreshing cache from master, [451ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [454ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [456ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [458ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [458ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [970ms] refreshing cache from master, [971ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [975ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [978ms] sending RPC to server e2c7b9c0f64b41c6b3f9a0e3084cf3e8, [980ms] received response from server e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [980ms] delaying RPC due to: Network error: Failed to connect to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127): Connection refused: /127.29.94.195:42127, [2490ms] refreshing cache from master, [2491ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [2495ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1514020439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@5c3e9abe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670) -> (continuation of Deferred@900035546 after retry RPC@1827379314) -> (continuation of Deferred@1127705915 after retry RPC@1940870439) -> (continuation of Deferred@1370158249 after retry RPC@1487246806) -> (continuation of Deferred@1754163160 after retry RPC@485573690) -> (continuation of Deferred@1631280916 after retry RPC@2004682653), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6419f2db -> wakeup thread Time-limited test -> (continuation of Deferred@1356916107 after retry RPC@2076461670) -> (continuation of Deferred@900035546 after retry RPC@1827379314) -> (continuation of Deferred@1127705915 after retry RPC@1940870439) -> (continuation of Deferred@1370158249 after retry RPC@1487246806) -> (continuation of Deferred@1754163160 after retry RPC@485573690) -> (continuation of Deferred@1631280916 after retry RPC@2004682653))) after lookup
14:06:39.004 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:39.005 [DEBUG - kudu-nio-7] (Connection.java:757) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] Successfully connected to peer
14:06:39.029 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0xf2417979, L:/127.0.0.1:59878 - R:/127.29.94.193:46741] using TOKEN/null
14:06:39.088 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.94.195:42127
14:06:39.088 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:06:39.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:39.089939 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:39.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:39.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:39.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data
14:06:39.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:06:39.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:06:39.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:06:39.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:39.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:39.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/logs
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.195:42127
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.195
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35565
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.195
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:39.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:06:39.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:39.438299 32372 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:06:39.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:39.438956 32372 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:06:39.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:39.439616 32372 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:06:39.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:39.475433 32372 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:06:39.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:39.476495 32372 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.195
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:39.514894 32372 tablet_server_runner.cc:78] Tablet server non-default flags:
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.195:42127
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.195
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35565
14:06:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.195
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/logs
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:06:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:39.516649 32372 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:06:39.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:39.518353 32372 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:06:39.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:39.532773 32378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:40.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:39.539152 32379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:40.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:39.542273 32381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:40.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:40.638445 32380 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
14:06:40.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.638480 32372 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:06:40.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.643069 32372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:06:40.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.645872 32372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:06:40.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.647429 32372 hybrid_clock.cc:648] HybridClock initialized: now 1783519600647374 us; error 76 us; skew 500 ppm
14:06:40.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.648313 32372 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:06:40.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.655035 32372 webserver.cc:533] Webserver started at http://127.29.94.195:35565/ using document root <none> and password file <none>
14:06:40.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.656035 32372 fs_manager.cc:362] Metadata directory not provided
14:06:40.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.656307 32372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:06:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.665366 32372 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.001s	sys 0.004s
14:06:40.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.671084 32391 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:06:40.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.672292 32372 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
14:06:40.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.672592 32372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:40.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:06:40.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:54 on dist-test-slave-59nb"
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.674685 32372 fs_report.cc:389] FS layout report
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/wal
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/data
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:06:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:06:40.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.734545 32372 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:06:40.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.736251 32372 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:06:40.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.736716 32372 kserver.cc:163] Server-wide thread pool size limit: 3276
14:06:40.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.739848 32372 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:06:40.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.746260 32398 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:06:40.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.754505 32372 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:06:40.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.754818 32372 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
14:06:40.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.755366 32372 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:06:40.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.761152 32372 ts_tablet_manager.cc:616] Registered 1 tablets
14:06:40.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.761358 32372 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.004s	sys 0.000s
14:06:40.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.761945 32398 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Bootstrap starting.
14:06:40.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.829638 32398 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Log is configured to *not* fsync() on all Append() calls
14:06:40.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:40.915866 32030 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c -> Peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127): Couldn't send request to peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.195:42127: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:06:40.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.930949 32398 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:06:40.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.932052 32398 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Bootstrap complete.
14:06:40.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.933480 32398 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Time spent bootstrapping tablet: real 0.172s	user 0.130s	sys 0.036s
14:06:40.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.953022 32398 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:40.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.955500 32398 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c7b9c0f64b41c6b3f9a0e3084cf3e8, State: Initialized, Role: FOLLOWER
14:06:40.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.956508 32398 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:40.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.960656 32398 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8: Time spent starting tablet: real 0.027s	user 0.029s	sys 0.000s
14:06:40.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.967661 32372 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.195:42127
14:06:40.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.967875 32508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.195:42127 every 8 connection(s)
14:06:40.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.971661 32372 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-2/data/info.pb
14:06:40.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:40.974620 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 32372
14:06:40.975 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:41.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.032660 32510 heartbeater.cc:344] Connected to a master server at 127.29.94.253:44767
14:06:41.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.033342 32510 heartbeater.cc:461] Registering TS with master...
14:06:41.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.035832 32510 heartbeater.cc:507] Master 127.29.94.253:44767 requested a full tablet report, sending...
14:06:41.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.042181 30196 ts_manager.cc:194] Re-registered known tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:41.043 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:06:41.043 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.94.193:46741
14:06:41.044 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:06:41.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.044559 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 32003
14:06:41.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.064873 32509 heartbeater.cc:344] Connected to a master server at 127.29.94.252:36353
14:06:41.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.065289 32509 heartbeater.cc:461] Registering TS with master...
14:06:41.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.066293 32509 heartbeater.cc:507] Master 127.29.94.252:36353 requested a full tablet report, sending...
14:06:41.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.069905 30274 ts_manager.cc:194] Re-registered known tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:41.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.072963 30274 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.94.195:54159
14:06:41.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.078992 32511 heartbeater.cc:344] Connected to a master server at 127.29.94.254:40897
14:06:41.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.079342 32511 heartbeater.cc:461] Registering TS with master...
14:06:41.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.079988 32511 heartbeater.cc:507] Master 127.29.94.254:40897 requested a full tablet report, sending...
14:06:41.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:41.084116 30120 ts_manager.cc:194] Re-registered known tserver with Master: e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:41.095 [DEBUG - kudu-nio-7] (Connection.java:239) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] handling channelInactive
14:06:41.096 [DEBUG - kudu-nio-7] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state READY due to: connection closed
14:06:41.097 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:41.097 [DEBUG - Time-limited test] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:41.099 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.099 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.099 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.121 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:41.121 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:41.121 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [3ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [23ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1191799526(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@d014a69 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d304f31 -> wakeup thread Time-limited test)) after lookup
14:06:41.121 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:41.123 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.123 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.123 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.140 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:41.140 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:41.141 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [3ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [23ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [24ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [26ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [26ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [43ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1191799526(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@d014a69 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1814436343 after retry RPC@1406175877), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d304f31 -> wakeup thread Time-limited test -> (continuation of Deferred@1814436343 after retry RPC@1406175877))) after lookup
14:06:41.141 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:41.143 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.143 [DEBUG - kudu-nio-2] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.143 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.159 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:41.159 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:41.160 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=62), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [3ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [23ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [24ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [26ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [26ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [43ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [44ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [46ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [46ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [62ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1191799526(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@d014a69 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1814436343 after retry RPC@1406175877) -> (continuation of Deferred@334044685 after retry RPC@1153051223), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d304f31 -> wakeup thread Time-limited test -> (continuation of Deferred@1814436343 after retry RPC@1406175877) -> (continuation of Deferred@334044685 after retry RPC@1153051223))) after lookup
14:06:41.161 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:41.162 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.162 [DEBUG - kudu-nio-3] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.162 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.240 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:41.240 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:41.240 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=143), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [3ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [23ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [24ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [26ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [26ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [43ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [44ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [46ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [46ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [62ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [63ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [65ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [65ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [138ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [142ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1191799526(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@d014a69 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1814436343 after retry RPC@1406175877) -> (continuation of Deferred@334044685 after retry RPC@1153051223) -> (continuation of Deferred@1111169468 after retry RPC@1464232766), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d304f31 -> wakeup thread Time-limited test -> (continuation of Deferred@1814436343 after retry RPC@1406175877) -> (continuation of Deferred@334044685 after retry RPC@1153051223) -> (continuation of Deferred@1111169468 after retry RPC@1464232766))) after lookup
14:06:41.241 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:41.243 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.243 [DEBUG - kudu-nio-4] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.243 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.700 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:41.700 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301),e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)[L],e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:41.701 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=604), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [3ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [23ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [24ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [26ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [26ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [43ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [44ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [46ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [46ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [62ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [63ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [65ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [65ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [138ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [142ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [144ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [146ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [146ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [599ms] refreshing cache from master, [599ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [602ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1191799526(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@d014a69 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1814436343 after retry RPC@1406175877) -> (continuation of Deferred@334044685 after retry RPC@1153051223) -> (continuation of Deferred@1111169468 after retry RPC@1464232766) -> (continuation of Deferred@61590338 after retry RPC@1733320594), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d304f31 -> wakeup thread Time-limited test -> (continuation of Deferred@1814436343 after retry RPC@1406175877) -> (continuation of Deferred@334044685 after retry RPC@1153051223) -> (continuation of Deferred@1111169468 after retry RPC@1464232766) -> (continuation of Deferred@61590338 after retry RPC@1733320594))) after lookup
14:06:41.702 [DEBUG - kudu-nio-0] (Connection.java:748) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] connecting to peer
14:06:41.703 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.704 [DEBUG - kudu-nio-5] (Connection.java:709) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] cleaning up while in state CONNECTING due to: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:41.704 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741) for tablet 2a6d316bea6f4d1ba1862ba16772c63b: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741
14:06:42.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.078017 32509 heartbeater.cc:499] Master 127.29.94.252:36353 was elected leader, sending a full tablet report...
14:06:42.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.514358 32521 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e1649b01489046e3a63ca563aad7869c)
14:06:42.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.514755 32521 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:42.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.516533 32521 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:42.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:42.520900 32211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:06:42.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:42.524286 32211 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:06:42.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.529182 32461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" is_pre_election: true
14:06:42.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.529842 32461 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957569291a7c402594d391a83985d059 in term 10.
14:06:42.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.530958 32211 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: e1649b01489046e3a63ca563aad7869c
14:06:42.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.531551 32521 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 11 FOLLOWER]: Leader pre-election won for term 12
14:06:42.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.531828 32521 raft_consensus.cc:493] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 11 FOLLOWER]: Starting leader election (detected failure of leader e1649b01489046e3a63ca563aad7869c)
14:06:42.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.532049 32521 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 11 FOLLOWER]: Advancing to term 12
14:06:42.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.536216 32521 raft_consensus.cc:515] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:42.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.537699 32521 leader_election.cc:290] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 12 election: Requested vote from peers e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741), e2c7b9c0f64b41c6b3f9a0e3084cf3e8 (127.29.94.195:42127)
14:06:42.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.539264 32461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a6d316bea6f4d1ba1862ba16772c63b" candidate_uuid: "957569291a7c402594d391a83985d059" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8"
14:06:42.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.539801 32461 raft_consensus.cc:3060] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 10 FOLLOWER]: Advancing to term 12
14:06:42.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:42.541313 32211 leader_election.cc:336] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111)
14:06:42.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.546123 32461 raft_consensus.cc:2468] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957569291a7c402594d391a83985d059 in term 12.
14:06:42.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.547057 32211 leader_election.cc:304] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957569291a7c402594d391a83985d059, e2c7b9c0f64b41c6b3f9a0e3084cf3e8; no voters: e1649b01489046e3a63ca563aad7869c
14:06:42.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.547721 32521 raft_consensus.cc:2804] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 12 FOLLOWER]: Leader election won for term 12
14:06:42.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.548738 32521 raft_consensus.cc:697] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [term 12 LEADER]: Becoming Leader. State: Replica: 957569291a7c402594d391a83985d059, State: Running, Role: LEADER
14:06:42.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.549589 32521 consensus_queue.cc:237] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:42.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.557667 30274 catalog_manager.cc:5697] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 reported cstate change: term changed from 11 to 12, leader changed from e1649b01489046e3a63ca563aad7869c (127.29.94.193) to 957569291a7c402594d391a83985d059 (127.29.94.194). New cstate: current_term: 12 leader_uuid: "957569291a7c402594d391a83985d059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } health_report { overall_health: HEALTHY } } }
14:06:42.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.962589 32461 raft_consensus.cc:1275] T 2a6d316bea6f4d1ba1862ba16772c63b P e2c7b9c0f64b41c6b3f9a0e3084cf3e8 [term 12 FOLLOWER]: Refusing update from remote peer 957569291a7c402594d391a83985d059: Log matching property violated. Preceding OpId in replica: term: 10 index: 36. Preceding OpId from leader: term: 12 index: 41. (index mismatch)
14:06:42.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:42.963907 32521 consensus_queue.cc:1048] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
14:06:42.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:42.986937 32211 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 -> Peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Couldn't send request to peer e1649b01489046e3a63ca563aad7869c. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:06:43.560 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2a6d316bea6f4d1ba1862ba16772c63b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638' with partition [<start>, <end>)
14:06:43.561 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2a6d316bea6f4d1ba1862ba16772c63b@[957569291a7c402594d391a83985d059(127.29.94.194:43301)[L],e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741),e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)]}]
14:06:43.561 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2a6d316bea6f4d1ba1862ba16772c63b, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2464), Traces: [0ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [2ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [3ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [23ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [24ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [26ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [26ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [43ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [44ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [46ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [46ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [62ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [63ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [65ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [65ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [138ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [142ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [144ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [146ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [146ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [599ms] refreshing cache from master, [599ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [602ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, [604ms] sending RPC to server e1649b01489046e3a63ca563aad7869c, [607ms] received response from server e1649b01489046e3a63ca563aad7869c: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [607ms] delaying RPC due to: Network error: Failed to connect to peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741): Connection refused: /127.29.94.193:46741, [2459ms] refreshing cache from master, [2459ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.94.252:36353, [2463ms] Sub RPC GetTableLocations: received response from server master-127.29.94.252:36353: OK, deferred=Deferred@1191799526(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fe0d8320c88@d014a69 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1814436343 after retry RPC@1406175877) -> (continuation of Deferred@334044685 after retry RPC@1153051223) -> (continuation of Deferred@1111169468 after retry RPC@1464232766) -> (continuation of Deferred@61590338 after retry RPC@1733320594) -> (continuation of Deferred@1264122621 after retry RPC@772899019), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d304f31 -> wakeup thread Time-limited test -> (continuation of Deferred@1814436343 after retry RPC@1406175877) -> (continuation of Deferred@334044685 after retry RPC@1153051223) -> (continuation of Deferred@1111169468 after retry RPC@1464232766) -> (continuation of Deferred@61590338 after retry RPC@1733320594) -> (continuation of Deferred@1264122621 after retry RPC@772899019))) after lookup
14:06:43.562 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] connecting to peer
14:06:43.563 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] Successfully connected to peer
14:06:43.591 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0x3396f808, L:/127.0.0.1:43672 - R:/127.29.94.194:43301] using TOKEN/null
14:06:43.637 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.94.193:46741
14:06:43.637 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:43.639567 30075 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/logs
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:06:43.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.193:46741
14:06:43.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.193
14:06:43.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:43.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35541
14:06:43.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.193
14:06:43.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:43.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:43.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:43.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:06:43.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:43.945895 32555 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:06:43.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:43.946439 32555 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:06:43.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:43.946946 32555 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:06:43.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:43.992995 32555 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:06:43.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:43.994378 32555 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.94.193
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:44.044953 32555 tablet_server_runner.cc:78] Tablet server non-default flags:
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.94.212:43443
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.94.193:46741
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.94.193
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35541
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.94.254:40897,127.29.94.253:44767,127.29.94.252:36353
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:06:44.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.94.193
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/logs
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 13:43:23 UTC on e7f111948823
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12775
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:06:44.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:44.046458 32555 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:06:44.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:44.048285 32555 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:06:44.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:44.060688 32561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:44.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:44.079035 32319 debug-util.cc:398] Leaking SignalData structure 0x7b08000be040 after lost signal to thread 32186
14:06:45.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:44.061465 32562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:45.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:45.421636 32564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:06:45.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:45.423877 32563 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1358 milliseconds
14:06:45.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.424002 32555 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:06:45.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.425541 32555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:06:45.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.428499 32555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:06:45.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.429950 32555 hybrid_clock.cc:648] HybridClock initialized: now 1783519605429862 us; error 81 us; skew 500 ppm
14:06:45.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.431033 32555 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:06:45.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.438714 32555 webserver.cc:533] Webserver started at http://127.29.94.193:35541/ using document root <none> and password file <none>
14:06:45.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.439879 32555 fs_manager.cc:362] Metadata directory not provided
14:06:45.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.440157 32555 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:06:45.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.450510 32555 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.005s	sys 0.000s
14:06:45.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.456277 32572 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:06:45.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.457584 32555 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
14:06:45.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.457996 32555 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data,/tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:45.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e1649b01489046e3a63ca563aad7869c"
14:06:45.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 14:05:51 on dist-test-slave-59nb"
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.460610 32555 fs_report.cc:389] FS layout report
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/wal
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/data
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:06:45.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:06:45.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:45.489988 32211 consensus_peers.cc:597] T 2a6d316bea6f4d1ba1862ba16772c63b P 957569291a7c402594d391a83985d059 -> Peer e1649b01489046e3a63ca563aad7869c (127.29.94.193:46741): Couldn't send request to peer e1649b01489046e3a63ca563aad7869c. Status: Network error: Client connection negotiation failed: client connection to 127.29.94.193:46741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:06:45.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.535888 32555 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:06:45.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.537693 32555 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:06:45.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.538208 32555 kserver.cc:163] Server-wide thread pool size limit: 3276
14:06:45.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.541353 32555 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:06:45.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.548442 32582 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:06:45.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.560292 32555 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:06:45.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.560640 32555 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.014s	user 0.000s	sys 0.003s
14:06:45.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.560976 32555 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:06:45.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.567020 32555 ts_tablet_manager.cc:616] Registered 1 tablets
14:06:45.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.567237 32555 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.006s	sys 0.001s
14:06:45.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.567802 32582 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap starting.
14:06:45.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.676054 32582 log.cc:826] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Log is configured to *not* fsync() on all Append() calls
14:06:45.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.827800 32582 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:06:45.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.828862 32582 tablet_bootstrap.cc:492] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Bootstrap complete.
14:06:45.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.830578 32582 ts_tablet_manager.cc:1403] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Time spent bootstrapping tablet: real 0.263s	user 0.192s	sys 0.050s
14:06:45.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.844599 32555 rpc_server.cc:307] RPC server started. Bound to: 127.29.94.193:46741
14:06:45.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.844731 32689 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.94.193:46741 every 8 connection(s)
14:06:45.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.848611 32555 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskB9LWx4/test-tmp/mini-kudu-cluster6292063065186127210/ts-0/data/info.pb
14:06:45.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.855160 30075 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu as pid 32555
14:06:45.856 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:06:45.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.866869 32582 raft_consensus.cc:359] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:45.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.870944 32582 raft_consensus.cc:740] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1649b01489046e3a63ca563aad7869c, State: Initialized, Role: FOLLOWER
14:06:45.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.891482 32582 consensus_queue.cc:260] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1649b01489046e3a63ca563aad7869c" member_type: VOTER last_known_addr { host: "127.29.94.193" port: 46741 } } peers { permanent_uuid: "e2c7b9c0f64b41c6b3f9a0e3084cf3e8" member_type: VOTER last_known_addr { host: "127.29.94.195" port: 42127 } } peers { permanent_uuid: "957569291a7c402594d391a83985d059" member_type: VOTER last_known_addr { host: "127.29.94.194" port: 43301 } }
14:06:45.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:45.919158 32582 ts_tablet_manager.cc:1434] T 2a6d316bea6f4d1ba1862ba16772c63b P e1649b01489046e3a63ca563aad7869c: Time spent starting tablet: real 0.088s	user 0.040s	sys 0.011s
14:06:45.936 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:06:45.983 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783519555638, tablet=2a6d316bea6f4d1ba1862ba16772c63b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:06:45.993 [DEBUG - main] (Connection.java:575) [peer master-127.29.94.252:36353(127.29.94.252:36353)] disconnecting while in state READY
14:06:45.994 [DEBUG - main] (Connection.java:575) [peer master-127.29.94.253:44767(127.29.94.253:44767)] disconnecting while in state READY
14:06:45.994 [DEBUG - main] (Connection.java:575) [peer master-127.29.94.254:40897(127.29.94.254:40897)] disconnecting while in state READY
14:06:45.994 [DEBUG - main] (Connection.java:575) [peer e1649b01489046e3a63ca563aad7869c(127.29.94.193:46741)] disconnecting while in state TERMINATED
14:06:45.995 [DEBUG - main] (Connection.java:575) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] disconnecting while in state READY
14:06:45.995 [DEBUG - main] (Connection.java:575) [peer e2c7b9c0f64b41c6b3f9a0e3084cf3e8(127.29.94.195:42127)] disconnecting while in state TERMINATED
14:06:45.998 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.29.94.253:44767(127.29.94.253:44767)] handling channelInactive
14:06:45.998 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.29.94.253:44767(127.29.94.253:44767)] cleaning up while in state READY due to: connection closed
14:06:45.998 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.29.94.254:40897(127.29.94.254:40897)] handling channelInactive
14:06:45.999 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.29.94.254:40897(127.29.94.254:40897)] cleaning up while in state READY due to: connection closed
14:06:45.999 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.29.94.252:36353(127.29.94.252:36353)] handling channelInactive
14:06:45.999 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.29.94.252:36353(127.29.94.252:36353)] cleaning up while in state READY due to: connection closed
14:06:45.999 [DEBUG - kudu-nio-6] (AsyncKuduClient.java:2704) Releasing all remaining resources
14:06:46.004 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] handling channelInactive
14:06:46.004 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 957569291a7c402594d391a83985d059(127.29.94.194:43301)] cleaning up while in state READY due to: connection closed
14:06:46.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:46.006053 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 32555
14:06:46.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:46.049903 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 32185
14:06:46.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:46.051330 32697 negotiation.cc:336] Failed RPC negotiation. Trace:
14:06:46.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.922175 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.29.94.193:32999 (local address 127.29.94.253:44767)
14:06:46.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.950865 (+ 28690us) server_negotiation.cc:207] Beginning negotiation
14:06:46.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.950958 (+    93us) server_negotiation.cc:400] Waiting for connection header
14:06:46.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:46.001699 (+ 50741us) server_negotiation.cc:408] Connection header received
14:06:46.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:46.049687 (+ 47988us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.94.193:32999: BlockingRecv error: recv got EOF from 127.29.94.193:32999 (error 108)
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":25758,"thread_start_us":26507,"threads_started":1}
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:46.051577 32692 negotiation.cc:336] Failed RPC negotiation. Trace:
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.862258 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.29.94.193:43255 (local address 127.29.94.252:36353)
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.902267 (+ 40009us) server_negotiation.cc:207] Beginning negotiation
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.902341 (+    74us) server_negotiation.cc:400] Waiting for connection header
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.902486 (+   145us) server_negotiation.cc:408] Connection header received
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.903067 (+   581us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.903143 (+    76us) server_negotiation.cc:462] Received NEGOTIATE request from client
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.904783 (+  1640us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.907321 (+  2538us) server_negotiation.cc:227] Negotiated authn=SASL
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.947518 (+ 40197us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.952356 (+  4838us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.956342 (+  3986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.957382 (+  1040us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:46.049813 (+ 92431us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.94.193:43255: BlockingRecv error: recv got EOF from 127.29.94.193:43255 (error 108)
14:06:46.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":38073,"thread_start_us":36715,"threads_started":1}
14:06:46.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 14:06:46.053968 32701 negotiation.cc:336] Failed RPC negotiation. Trace:
14:06:46.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.961040 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.29.94.193:36389 (local address 127.29.94.254:40897)
14:06:46.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.974392 (+ 13352us) server_negotiation.cc:207] Beginning negotiation
14:06:46.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:45.974472 (+    80us) server_negotiation.cc:400] Waiting for connection header
14:06:46.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 14:06:46.052706 (+ 78234us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.94.193:36389: BlockingRecv error: recv got EOF from 127.29.94.193:36389 (error 108)
14:06:46.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":11214,"thread_start_us":10521,"threads_started":1}
14:06:46.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:46.083061 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 32372
14:06:46.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:46.116986 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 30088
14:06:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:46.150602 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 30158
14:06:46.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 14:06:46.185374 30075 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskB9LWx4/build/tsan/bin/kudu with pid 30233
14:06:47.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T14:06:47Z chronyd exiting

Time: 68.508

OK (1 test)

